Everything always comes back around on How To Get Away With Murder, doesn't it? So far in Season 2, we've seen Annalise's big old pregnant belly, we've seen her working on the Wallace Mahoney case, we've seen her with a young Frank, and we've seen her get into a really awful car wreck. The season finale finally connected all of the dots of these seemingly unrelated events: Frank helped set up Annalise for Wallace Mahoney, who then ordered a hit on Annalise. So, in a way, Frank was responsible for Annalise's baby dying.
Let's back up for a moment — through flashbacks, we saw that a woman with a big bag of money previously approached Frank in a bar, propositioned him (for sex), and then propositioned him to bug Annalise's room. As it turns out, she was working for Wallace Mahoney, who wanted to listen in on Annalise's conversations to see what she was planning in his son's murder trial. Once Wallace discovered that Annalise knew he was Wes' dad and planned to tell people about it, he ordered the car crash to try and kill her. Though she survived the wreck, her unborn baby died soon after. Therefore, Frank is actually responsible for killing Annalise's baby, a boy who would have been named Sam, after his father.
But wait, it gets worse. Frank also told Sam about his role in the whole thing, and Sam forbade him from telling Annalise. When Frank killed Lila on Sam's orders 10 years later, he did it because Sam had something over him. Of course, he could never have known what would happen to Annalise and her baby, but that's a pretty big burden for Frank to hold on his shoulders. It makes sense that Frank just bailed on everyone — all of the guilt simply became too much for him to bear. Will he be back next season? You never know on How To Get Away With Murder.