What I Wish I'd Known About SOC2 Compliance: One Founder's Take

We have an exciting piece of news to announce today: Libretto has achieved SOC2 Type 2 compliance! This means that our customers can be assured that we uphold a certain set of industry-accepted security practices, and that an auditor has double checked our work. It’s an important step in our growth, and we expect it to help customers understand how we uphold security principles here at Libretto.

When they achieve SOC2 compliance, most companies put out an anodyne, bloodless blog post about upholding corporate excellence through process improvement, or something like that. I’d like to step back for a second, though, and write a somewhat more candid post about what I wish I’d known before we started on the SOC2 journey. 

First, my snarky, glib, and totally unfair take in Slack when we got the news that we got our SOC2 report:

‍

This, of course, isn’t fair, but at times, SOC2 compliance can feel like this. Let’s dig deeper into the good, the bad, and the ugly of SOC2 compliance, and what I wish I knew when I started.

Why Get SOC2 Type 2 At All?

First off, know this: if you’re selling software as a service, there are a lot of enterprise customers who just won’t talk to you if you don’t have SOC2 Type 2 compliance. It’s not a sufficient condition to get deals, but it’s often a necessary condition. Also, there’s a much easier version of SOC2 to get, called SOC2 Type 1, but my understanding is that most serious customers don’t put much stock in it. SOC2 Type 1 involves an auditor just checking that you have policies in place at one single point in time, whereas SOC2 Type 2 involves an auditor watching you over the course of an audit window to make sure that you actually follow your policies as events arise.

The other reason to get SOC2 Type 2 is that it does genuinely make you get a bit more formal about security processes and reduces the risk that you will have a major security fail. This is an important result, but I think that the vast majority of startups don’t get SOC2 Type 2 for this reason. They get it because of customers.

How Long Does It Take?

Second, understand that SOC2 can take a long time to achieve, so the earlier you start, the better. There are essentially three phases for getting SOC2: preparation, audit window, and audit report drafting. 

First, the preparation phase involves establishing SOC-compliant policies for a ton of different domains (physical security, information security, access control, human resources, data breaches, etc) and working to implement them. We did our preparation in about a month last August, and I did almost all the work myself. We made a flurry of changes to our processes and systems, things like formally requiring code reviews on every code check-in to main, auditing every change to access control of any system, requiring multi-factor auth on every system, and installing monitoring agents on all employee machines. 

I highly, highly, highly recommend using a SOC2 compliance platform when going for SOC2. Compliance platforms are like checklists on steroids. They list all the tasks you need to do, and in many cases they can automatically check your infrastructure to make sure it is in compliance. These platforms are expensive, but I genuinely can’t imagine successfully getting to compliance without one. (We used Vanta as our compliance platform, but the other major products look like they would get the job done as well.) When choosing a compliance platform, I would advise you to purchase the compliance platform that integrates with the most number of systems that you use. When your compliance platform integrates with your systems automatically, it can prove to your auditor programmatically that you are in compliance. When you don’t have direct integration, you have to take screenshots of your systems to prove to the auditors that your systems are in compliance, which rapidly becomes quite annoying.

You also have to choose and hire an auditor during your preparation phase. Your compliance platform can set up meetings with auditors for you, but honestly, it’s very hard to distinguish one from another. It’s also an intensely weird situation to be interviewing a company that in theory is going to be watching over you. In my experience, some auditors will subtly indicate that they will move more quickly and maybe will not look as closely as other auditors, which made me feel uncomfortable and made the whole process feel like it had a lot less integrity than one would hope. In the end, I went with the auditor that seemed the most buttoned up and formal, under the theory that if I was going to do this, I should do it right. I’m not sure if that was the right call, as it probably added some time to the report creation phase, but I’m proud that we approached the process with integrity.

Next, the audit window is the official period where the auditors are watching you to make sure that you follow your policies and that your organization follows SOC2 guidelines. The first time you do an audit, the minimum audit window you are allowed to have is 3 months, and we did ours from September through November of last year. During this time, you can finish up implementation of some of your policies, and you should talk with your auditors frequently about what is and isn’t yet done. Confusingly, there are some SOC2 tasks that you have to do during the audit window, and I ended up having to repeat some tasks that I had already finished during the preparation phase. For this reason, it’s a good idea to hire your auditor early in the preparation phase so you can ask them questions.

During the audit window, you will need to hire a penetration testing company to perform a penetration test on your systems. Often, your auditor will have a pen testing service that they offer, but we went with a separate security company. This again is a strange situation, where you are hiring your own watchdog, and I found that pen testing companies tended to pitch themselves as either quick or comprehensive, with the implication that the quick companies might go a little easier on you. Again, we went with the company that seemed most comprehensive, Mirai Security, and I was really happy with their work. They found one pretty great bug in our system, which turned out to be a more general bug in Next.js, so they deserve a lot of kudos. Pen tests almost always find a few issues, and usually you will build in a retest from the pen testing company after you have fixed the problems. It’s important to build in the fixing and retest time to your audit window; it’s very important that you end up with a clean retest before your audit window ends.

The two pieces of advice I would give during the audit window are don’t procrastinate and ask your auditor a lot of questions. It’s typical to procrastinate tasks to the end of the audit window, and this can cause a crunch that will randomize your team. We did reasonably well at avoiding this, but I will admit that both of the tabletop exercises happened in the last week of our audit window. The reason to ask your auditor lots of questions during the audit window is that they won’t generally tell you pro-actively if you’re messing something up, but they will usually be able to tell you if you ask them. Given that the rules around SOC2 are somewhat opaque, it always pays to ask the auditor any questions or confusion that comes up as you’re working through the checklists. I ended up asking a bunch of questions about how the auditor would think about the software platforms that we use (like Vercel and Timescale), because Vanta doesn’t really deal with those platforms very well. I’m sure that if I hadn’t asked those questions, we would have had a harder time getting certification. And in one case (which I’ll talk about below), I failed to ask a question that would have made the certification process a lot better.

Finally, during the report creation phase the auditor gets to work writing your audit report. During this phase, they will use your compliance platform to check most of your policy adherence but will also ask you for further proof of things they need to drill down on. This phase took quite a while for us: three months. My understanding is that it usually takes more like half that. This is where you will learn if there are any exceptions in your report (more on that below), and you will eventually get your fully certified audit.

How Much Does It Cost?

Startups are always looking to minimize burn, and there isn’t a lot of transparent information about how much getting SOC2 costs. The big ticket items are a compliance platform, a penetration test, and an auditor. Talk to all of the major compliance platforms to get quotes, and definitely try to negotiate them against each other.

We found that, for tiny startups, compliance platforms tend to cost about 6 to 8 thousand per year, penetration tests cost 5 to 7 thousand, and auditors cost 5 to 8 thousand. There are also some miscellaneous fees, like $50 per employee for background checks. I’d recommend budgeting somewhere around $20k to $25k for your first SOC2 Type 2 audit, although if you are aggressive in negotiation, you might be able to get it down a bit.

In terms of workload, I’d estimate four to six weeks of someone’s time all told is required for the compliance journey, though that probably is heavily dependent on what kinds of technologies you have in your system.

If you’re trying to get SOC2 on a shoestring budget, it’s worth noting that while a penetration test and an auditor are absolutely required costs, technically a compliance platform like Vanta is not. However, it is basically impossible for me to believe that anyone could successfully complete SOC2 Type 2 without a compliance platform tracking progress and automatically testing for issues. There are far, far too many gotchas and opaque conventions in SOC2; if you go in without a compliance platform, the chances that you mess something up as a non-expert are near 100%.

What’s Good And What’s Annoying About SOC2 Certification

So, aside from the fact that customers want to see a SOC2 badge, is SOC2 good or bad for your startup?

While there’s a ton of busy work in SOC2 that is designed for larger companies, I do think that startups can benefit from many of the SOC2 tasks. Speaking frankly, security at many startups is way, way, way too lax, and SOC2 makes you button up processes and systems in ways that can genuinely help keep your customers’ data secure. Specifically, I like that we at Libretto are now more formal about access control, that we require code reviews for every check-in, and that we have multi-factor turned on everywhere. The audit also made us be much more systematic with our incident response policy, including making us do a tabletop disaster recovery exercise and a tabletop security incident response exercise, which ended up being both fun and revealed some things we could do better.

There are, unfortunately, also a ton of box-checking exercises when it comes to SOC2: an endless stream of AWS configuration changes, for example, some of which frankly make little sense and are unlikely to help us to meaningfully be more secure. It’s also the case that SOC2 auditors and platforms tend to assume that you control your entire infrastructure on AWS or Google Cloud and can sometimes be confused if you use PaaS platforms, as we do here at Libretto. There are definitely some one-size-fits-all tests that compliance platforms and auditors use, and those cause confusion and wasted effort.

The Dreaded Exception

Speaking of confusion and wasted effort, I want to talk a bit about what can go wrong in the audit process, and specifically what went wrong for us.

I should first say that the goal of the SOC2 process, from your perspective as a founder, is to get an unqualified SOC2 Type 2 report without exceptions. The first part, unqualified, is the overall “grade” you got from the auditor, and there are four possibilities here:

• Unqualified means the auditor thinks you “passed”.
• Qualified means that you came close, but the auditor has some reservations about your processes.
• Adverse means you didn’t come close to passing.
• Disclaimer of Opinion means the auditor didn’t get enough information to make a determination about whether you passed or not.

In addition to the overall grade, you can get “exceptions” on your report. Exceptions are specific instances of moments where you failed to follow a SOC2 policy. Just because you get an exception doesn’t mean you will fail your SOC2 report, or even that you will get a qualified report. It all depends on how serious the exception was.

Unfortunately, we ended up with an exception in our report, but how it happened is instructive, so I’m going to share the story.

One of the requirements of SOC2 is that you have and follow a policy that requires yearly performance reviews. We adopted such a policy in August 2024 and planned to do our first review cycle in November 2024. When it came time to do them, we were busy and asked our auditor if we could put them off until December 2024; he said that’d be fine. Then in December, we were still busy, so I figured it wouldn’t be a problem to put them off until Q1 2025. We then did the reviews at the beginning of February. I had assumed that since we’d had our yearly performance review policy for only 7 months at that point (and since it was presumably our choice when to do reviews) that this would be no problem. But later in February our auditor let us know that we would get an exception because we hadn’t done performance reviews in the previous calendar year.

To be honest, I was annoyed at this, both at myself and at the auditor. I was annoyed at myself because it wouldn’t have been that hard to get the reviews done in December, and I failed the team by not asking more questions to make sure that moving our reviews to Q1 2025 would be a problem. But I also was somewhat annoyed with the auditor for not pointing out, when I asked about moving the reviews to December, that moving the reviews any further would cause problems.

In the end, it’s not that big of a deal. We’re a young company, so there were only 2 employees who went longer than a year without a review, and the longest time an employee had without a formal review was 15 months. Our SOC2 report is unqualified. And one nice thing is that you get a chance to explain the circumstances around an exception in your SOC2 report, and I think our explanation is compelling. Having the exception at all could raise flags with customers who follow policies strictly, though.

The lesson here is, again: don’t procrastinate SOC2 tasks and ask lots of questions to your auditor.

Summing Up

Hopefully, this has been a useful look into what SOC2 Type 2 compliance can look like for a small startup. I want to thank our auditors Insight Assurance, our penetration test provider Mirai Security, and the folks over at Vanta; without the help from each of those organizations, we wouldn’t have gotten over the finish line on this. I’d also like to thank my co-workers for putting up with my endless requests for their help in making sure we improve and uphold our security values.

If you have any questions, please feel free to reach out, and if you have any need for a tool to help you monitor, test, and optimize LLMs in your product, definitely consider using SOC2-compliant Libretto!