In his biography of Thomas Jefferson: The Art of Power by Jon Meacham, Meacham discusses Jefferson's time in France as ambassador and that he took Sally Hemings with him. When it was time to come back to America, she could have readily refused and the French would have allowed her to stay.

That's not to say she WANTED to go back with Jefferson. Meacham suggests that Hemings made some sort of agreement with Jefferson in exchange for going back to Virginia. Likely that he must free her children as they came of age.

According to Monticello.org: Decades later, Jefferson freed all of Sally Hemings's children – Beverly and Harriet left Monticello in the early 1820s; Madison and Eston were freed in his will and left Monticello in 1826. Jefferson did not grant freedom to any other enslaved family unit. As to Sally herself, Jefferson did not officially free Sally Hemings but his daughter Martha Jefferson Randolph permitted her to leave Monticello shortly after Jefferson's death in 1826. Sally went to live with her sons Madison and Eston in Charlottesville

And you are absolutely right- it's not an affair if one can't consent!!