"Is it not my right to tell the true story?" We're not sure exactly what you mean by that. There is no basic legal right to tell one's life story (although a lot of people using cell phones in public seem to think so). There is as you know, a right to speak freely. however, but even that right comes with some caveats.
Defamation and Invasion of Privacy. There are two major free speech exceptions when writing memoirs and autobiographies. (We got asked the difference the other day and that led to an interesting discussion.) Those exceptions have to do with (1) whether you are defaming someone (telling lies that cause harm) or (2) invading someone's privacy. Invasion of privacy comes in several flavors but generally, authors are protected as long as they are providing fact-based recountings of actual accounts.
Public figures and defamation ... Legally, it is "harder" to defame a public figure than a regular civilian. That's because public figures are supposed to have thicker skins (must be from all that spotlight heat) and a wronged celebrity would have to prove actual malice, a higher hurdle than what's required for mere mortals.
Your author contract. If you sell your memoir to a publisher, your author contract will likely require that you defend the publisher against any claims that occur. That could be risky from a personal liability point of view ... so in that case, you may want to have the book vetted for issues like this by a media attorney.