If you have been watching The Crown, Netflix’s hit series about the life of Queen Elizabeth, you may have been surprised to see American royalty visit with British royalty, when Jackie and John F. Kennedy stopped by Buckingham Palace. Like much of the show, the story was based on real history.

In June 1961, just months after JFK started his presidential term, the Kennedys paid a visit to the Queen, bringing a signed portrait of the president in a silver Tiffany’s frame, with a message he had handwritten: “To Her Majesty Queen Elizabeth II, with appreciation and the highest esteem, John F. Kennedy.” The Queen threw a banquet in their honor, which Prime Minister Harold Macmillan described in his diary as “very pleasant.” In a birthday note to the Queen, JFK himself remembered the festivities fondly. “May I also at the same time say how grateful my wife and I are for the cordial hospitality offered to us by your Majesty and Prince Philip during our visit to London last Monday,” he wrote. “We shall always cherish the memory of that delightful evening.”

However, rumors have persisted about the evening ever since it took place, which were played up for drama’s sake in the episode from season two of The Crown. In the show, Queen Elizabeth (played by Claire Foy) gives Jackie a personal tour of the palace, and the two women discuss what it’s like to be public figures who aren’t particularly fond of the spotlight. While it seems like the two icons are bonding, later in the show Jackie says some unkind things about the Queen and Buckingham Palace, going so far as to call the royal family’s home “second-rate, dilapidated and sad, like a neglected provincial hotel.” When Jackie’s words were reported to her, Elizabeth, witty as ever in the show, responds by saying, “Well, we must have her again soon.”

While The Crown may have exaggerated the exchange, according to Jackie’s confidantes, Cecil Beaton and Gore Vidal, Jackie did have some criticisms of Queen Elizabeth following the actual 1961 meeting. Per The Telegraph, Beaton claimed that Kennedy “was unimpressed by the palace furnishings and by the Queen’s dress and hairstyle.” And, according to Reader’s Digest, Jackie supposedly told Vidal, “I think the queen resented me. Philip was nice, but nervous. One felt absolutely no relationship between them.”

According to The Telegraph, that wasn’t all. Vidal also claimed that after their meeting, Jackie described Elizabeth as “pretty heavy going.” Vidal mentioned the comment to Elizabeth’s sister, Princess Margaret, years later, she is is said to have retorted, “But that’s what she’s there for.” In The Crown, Jackie apologized for speaking unkindly about the Queen, blaming her loose lips on medication, but it’s unclear whether that happened in real life—or even if the Queen knew about Jackie’s disrespect. What history tells us, though, is that when Jackie was in London the following year, the Queen invited her to lunch and Jackie later told the press that she was “grateful” for the invitation and found the Queen “charming.” The friendship seemed to recover from the rumors. When JFK was assassinated in Dallas in 1963, Prince Philip attended the funeral. Later, Queen Elizabeth opened a memorial dedicated to JFK in England and Jackie and her children attended the ceremony.