From Shirley Jackson's essay, "Experience and Fiction":

"[M]y husband and I . . . were on the train which stops briefly at the 125th Street station, and just outside the station, dim and horrible in the dusk, I saw a building so disagreeable that I could not stop looking at it; it was tall and black and as I looked at it when the train began to move again it faded away and disappeared. . . . From that time on I completely ruined by whole vacation in New York City by dreading the moment when we would have to take the train back and pass that building again. . . . [E]ven after we were home it bothered me still, coloring all my recollections of a pleasant visit to the city, and at last I wrote to a friend at Columbia University and asked him to locate the building and find out, if he could, why it looked so terrifying. When we got his answer I had one important item for [The Haunting of Hill House]. He wrote that he had had trouble finding the building, since it only existed from that one particular point of the 125th Street station; from any other angle it was not recognizable as a building at all. Some seven months before it had been almost entirely burned in a disastrous fire which killed nine people. What was left of the building, from the other three sides, was a shell. The children in the neighborhood knew that it was haunted."

According to Ruth Franklin, the Hill House author exaggerated the details of her account. In Shirley Jackson: A Rather Haunted Life, Franklin writes:

"Like so many writers' best stories, this one seems only to be partially true. There was a fatal fire in Harlem in April 1957, in which three people (not nine) were killed and five injured; but the apartment building where it took place was at 229 West 140th Street, too far to be seen from the 125th Street station. If Jackson received the letter she describes from a professor at Columbia, no record of it exists in her files. But she may well have glimpsed an uncanny-looking apartment house from the train that got her thinking about how houses become haunted."

From "Experience and Fiction":

"I was collecting pictures of houses, particularly odd houses, to see what I could find to make into a suitable haunted house. . . . I came across a picture in a magazine which really looked right. It was the picture of a house which reminded me vividly of the hideous building in New York; it had the same air of disease and decay, and if ever a house looked like a candidate for a ghost, it was this one. All that I had to identify it was the name of a California town, so I wrote to my mother, who has lived in California all her life . . . Yes, she knew about the house, although she had not supposed that there were any pictures of it still around. My great-grandfather built it. It had stood empty and deserted for some years before it finally caught fire, and it was generally believed that that was because the people of the town got together one night and burned it down."

The house Jackson spotted was a Nob Hill mansion belonging to Charles Crocker, and was built by her great-grandfather, Samuel Bugbee. The house existed at the center of a long property dispute between Crocker and his neighbor, Nicholas Yung, during which time Crocker built a 40-foot-tall spite fence around Yung's land. The Crocker Mansion was destroyed in the early 20th century, but some parts of the fence remained. The site is now the home of San Francisco's Grace Cathedral.

Although many sources point to Bennington College's Jennings Music Building — the next item on this list — as the inspiration for Jackson's haunted house, Franklin disagrees:

"Jackson's files contain no picture of [Jennings], and the house is much plainer than the others in her collection. A better local candidate is the Edward H. Everett mansion near Old Bennington, which at the time was being used as the novitiate of the Holy Cross Congregation and is now part of the campus of Southern Vermont College. . . . It is said — still — to be haunted by the ghost of Everett's first wife, a woman dressed in white who roams the house and grounds."

Franklin may not believe that the Jennings Music Building inspired Jackson's titular Hill House, but students at Bennington College, where the author's husband taught in the 1950s, do. The college's head music librarian told Cosmopolitan earlier this year that "the basement [of the music building] is very creepy . . . That's the one place I'm scared to go to, even during the day. It's a long, dark corridor. It has safety lights, but there's a feeling of darkness . . . There's all these little caged spaces and locked doors, where faculty used to store items."

Sounds pretty haunted to me.

The site of the Moberly-Jourdain incident, as chronicled in Charlotte Anne Moberly and Eleanor Jourdain's 1911 book, An Adventure, Petit Trianon was originally the home of Louis XV's mistresses, and was later the private estate of Marie Antoinette. An Adventure recalls a 1901 trip to Versailles in which the two authors claim to have entered a time slip that transported them back to 18th century France, where they met various servants and Marie Antoinette herself.

For Moberly and Jourdain's strange encounter, there may have been a more logical explanation than time travel, however. From The Telegraph:

"It is possible that the ghosts may have been real people wearing 18th-century costume, dressed up either for a fête champêtre or for a film, or perhaps as guests at one of the parties which were known to have been held around this time at the Petit Trianon by the dandyish Comte de Montesquieu. But there is no conclusive evidence for any of this."

Either way, the story seems to have stuck with Jackson. In Shirley Jackson: A Rather Haunted Life, Franklin writes, "In her notes for [Hill House], Jackson refers to An Adventure as 'one of the greatest ghost stories of all time.'"

Finally, there's the Winchester Mystery House, which got its own horror movie in early 2018. The house has a colorful history, having been built and rebuilt over a period of nearly 40 years until the death of its owner, Sarah Winchester, in 1922. She was the widow of William Wirt Winchester, the son of Winchester rifle creator Oliver Winchester, and the treasurer of the Winchester Repeating Arms Company.

According to legend, Sarah Winchester believed that her house was haunted by the ghosts of people killed by her in-laws' rifle company, and that the only way to outwit the spirits was to build a house so confounding in its design that they could not navigate it. From Shirley Jackson: A Rather Haunted Life:

"A medium reportedly told [Sarah Winchester] that spirits were taking revenge on her for all the people who had been killed by Winchester firewarms. The only solution was to build a house that was in flux: as long as rooms, corridors, and stairs were constantly added, the medium said, the confused spirits would not attack. . . . [The Winchester Mystery House's] idiosyncratic deisgn includes gables and turrets bursting out at all angles, corridors at lead nowhere, stairs with uneven risers, trapdoors, and ornamentation featuring spiderwebs and the number thirteen (Winchester believed both had spiritual significance). Hallways are as narrow as two feet; doors open both inward and outward in unexpected places, with some on the upper floors leading directly to a sudden drop. . . . Hill House would incorporate some very similar features, including the disorienting layout, and the novel mentions the Winchester House."

Sounds like a pretty great place to set a ghost story, right?