QuickBooks Data Corruption is a Common Problem

The Truth about QuickBooks Desktop Files

Whether you’ve been using QuickBooks™ for years or just a just a short time you probably noticed all the updates and versions Intuit releases. With all of that activity and attention you trust that your data is in good shape. After all, people have been relying on QuickBooks since 1992.

Well, it’s time to tell the truth about the health of your QuickBooks company file.

In a perfect world, your software and data would work smoothly, doing precisely what they are designed to do. Unfortunately, we live in a dynamic world–both natural and digital–and things degrade, no matter how cautious you think you are. It’s called entropy.

Simply defined, entropy is a gradual decline into disorder

Visualization of Entropy

 

 

 

 

There are four factors that affect the decline – and performance – of your file:

  1. File Size
  2. The number of DB File Fragments
  3. The earliest of the Versions Used on File
  4. Daily use and bad user habits

To locate details on the first three of these items for your own QuickBooks Desktop company file, hit the F2 key once the file is open. You’ll see the Product Information window. (In Figure 1, I’ve highlighted the items I’m referring to.)

Product Info

Figure 1. Open QuickBooks and hit F2 on your keyboard to see the Product Info screen

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Let’s review these factors, and the actions you can take to protect your data against corruption and loss.

There is Such a Thing as Too Big

From our years of hands-on experience working with hundreds of QuickBooks files, we have established real-world limits on the sizes of QB files before corruption and performance become serious issues:

Pro:                        250 MB
Premier:              350 MB
Enterprise:         1.25 GB

Intuit, the maker of QuickBooks, seems to have a different take on file size. For instance, some support agents have told our customers that Enterprise can be 4GB or larger. Perhaps in a sterile, carefully tended environment. But that’s not the way small businesses run. We currently have a Pro customer whose file is almost 700 MB. It’s incredibly slow to open and to load into the data warehouse we use. For two years I’ve recommended that they start a new file. Now they are asking if integrated applications can sync more quickly. We design miracle reports, but we can’t help with that miracle!

If your company file is close to, or has exceeded, the maximum file size for your version, there are three options for you to take:

  1. File Split. If your file is otherwise healthy, we can refer you to Advanced Certified QuickBooks ProAdvisors who specialize in eliminating old data. It’s often faster and less expensive than creating a new file. This is probably not an option, though, if you use Advanced Inventory or Pricing Levels.
  2. New File Creation. If your file has corruptions, is older than 10 years, or a File Split will only create a temporary reduction, you will want a new file created. I cannot stress enough how critical it is to hire an experienced ProAdvisor to perform this project, as it’s an involved project. I have a colleague who wrote a 57-page document explaining most of the process. It really needs to be a coordinated effort.
  3. Do Nothing. If your file is relatively healthy, you can temporarily do nothing. This is temporary, however, and you risk the loss of some or all of your data, even if you do regular backups.

As a fellow business owner, I urge you to take this caveat very seriously. I have had a customer call me in a panic after receiving a scary error message from QuickBooks. The file was kaput and the only option was to start from scratch and re-enter months’ worth of transactions.

Issues Beyond File Size

For most businesses, file size isn’t the most common problem. In fact, it’s often just a symptom. The problem is actually simple: you’re using it. Opening and closing a file can lead to degradation and entropy. Software glitches, power surges, and just sharing or copying a file to a new location can cause corruption.

One source of QB file corruptionsLet’s say you’ve been using QuickBooks for quite some time. In 10 years, having just 3 users who are opening and closing a file twice per day (because you know… it crashes sometimes) means there were over 15,000 opportunities for something to go wrong.

If your file is larger than the sizes already mentioned, your file is at greater risk because it runs more slowly. More time opening, saving, and closing means more exposure to the problems previously discussed here. If we include bad user habits, like not exiting at the end of the day, leaving transactions open and incomplete, we see even more corruptions and broken links.

One indicator for this kind of issue is the number of DB File Fragments. See Graphic 1. File Fragments are when QuickBooks has started saving packets of data in non-adjacent space on the hard drive. The more spread out the related packets are, the slower the performance of QuickBooks. There is a simple way to reduce or eliminate the file fragments – see the section at the end of this article.

As you use QuickBooks on a day-to-day basis, you likely will not notice there’s a problem. Sure, it may glitch or crash now and then, but it starts right back up and your transactions are still there. But entropy is real, folks, and without implementation of solid data policies, your data could be at risk.

Fortunately, you can take steps to protect your data.

Implement Data Policies

Investigate the topic of data policies. Plenty of accounting systems don’t allow you to keep more than two or 3 years of historical data in the active file. Before QuickBooks came along, every accounting solution gave you two years of data MAX, and you had no choice – and that included Peachtree and Great Plains. You can always keep your old file for reference purposes. Something worth considering. I promise you – you do not need 25 years of accounting data in your active file.

I just have to share something I saw yesterday – one of our customer’s files has been in use since it was a Quicken file. And it’s 1.3 GB. You can bet that the owner of the company is going to get a strongly worded invitation in the very near future to get a new file created.

Early Detection and Correction

QuickBooks does have data verification and rebuild tools built in. These tools are easy to use–just access the File | Utilities menu. The problem is that oftentimes the software will report that “QuickBooks detected no problems with your data” and then you might believe it. Some customers run a “Verification” thinking that will do something to the file – but it doesn’t, and it doesn’t always report problems when the exist.

All of that makes the job of a trusted advisor much more difficult. Some customers don’t take our warnings seriously, or they ignore them altogether.

Age is Just a Number … Or is it?

In 2006, Intuit made a significant change to the data structure and the way it is accessed, while still accommodating the way existing customers had been using the software. This means there may be data in fields where it no longer belongs because it’s a different data type. For example, there may be text in a field designed for phone numbers. If your file has been in use since 2006 or earlier, your data could be at risk. In this case we recommend a New File Creation. We found a customer last week who has been using their file since it was a Quicken file. No joke.

Why File Care is Important

There are many reasons file care is important – but the biggest is that you, the customer, can’t tell it’s happened.

Data Corruption - Image

In 2013, I discovered a tool called QQube that makes it easier to access the data in QuickBooks. It uses the software development kit (SDK) and the Custom Reporting function to extract data from the system. The SDK sniffs out data corruption, never lies about it, and simply won’t transfer the data out. Initially, I was frustrated by this. But after finding so many files that had otherwise undetectable issues, I realized it’s the only early warning system we’ve got.

Your QuickBooks users can help with file care every day. This means always closing all open windows before exiting the program entirely. Many users who access QuickBooks via Remote Desktop (formerly known as Terminal Services) only close the Remote Desktop window – and forget to exit out of QuickBooks. This can cause issues.

Another cause of corrupted files is weak or faulty network connections – which can be caused by a loose or broken Ethernet cable or intermittent WiFi connections. These dodgy connections interrupt the process for saving transactions.

You may have noticed that QuickBooks sends out frequent fixes and updates. With every release, there are changes under the hood. When you do not establish a strict software update policy and process, QuickBooks allows users to continue using older releases. This, too, can cause problems.

The Real Cost of QuickBooks

Selecting an accounting system to suit your operational needs­–and your budget–is obviously critical. When you start out small, QuickBooks can usually handle the requirements. The problems start once significant growth begins – or when no one is tending to file care.

It’s the choice between pay now – or really pay later. In our view, if you opt to use QuickBooks, you’re choosing to pay later. What you’ll pay for is a new file creation or file reduction every few years, to the tune of $2,000 to $20,000, depending on how much historical data you truly need to keep in your production file.

Practical Prevention

QuickBooks Desktop is an attractive, affordable option. But you need to understand how to care for your valuable data. It simply is not “set it and forget it.” Here are tips you can implement to help protect your business:

  • Set and follow a regular Rebuild schedule
    • Read here for excellent instructions and best practices for a Rebuild
    • Understand that running a “Verify” doesn’t do anything to the file – and may not detect significant issues
  • Perform a “Portable Copy Round Trip” to reduce the DB File Fragments to under 10
  • Don’t use the Condense feature – except, perhaps, on the Audit Trail if you’re using it
  • Delay upgrading to the newest year of QuickBooks until around 6 months after its release
  • Create and adhere to an update policy that:
    • Disallows automatic updates
    • Only allows IT to perform the update
    • Updates everyone on the next login
  • Don’t assume your IT provider knows how to manage and maintain QuickBooks
  • Respect the minimum hardware specifications, especially for RAM, processing speed, and disk size
  • If your file is larger than recommended in this article or older than 10 years, bite the bullet, engage a pro, and get a fresh new file created

We Love QuickBooks, But…

It seems like Intuit doesn’t instruct end users how to care for their files until there is a problem. And very often, that’s too late and the problem cannot be resolved. While I appreciate that we live in a dynamic world, I consider this to be unethical. A few years ago Intuit ran a QuickBooks campaign with the slogan, “Own It.” In my opinion, if you’re going to own QB, you’d better be willing to own the limitations that accompany the very reasonable pricing. It’s my hope that Intuit owns it, too, by:

  • Telling customers the truth about file limitations
  • Instructing customers how to care for their QuickBooks files
  • Removing the condense feature
  • Disallowing the use of multiple releases on the same file and network

Most business owners, and even many Certified QuickBooks ProAdvisors, are genuinely surprised to learn there’s a problem with the files. After all, it looks fine, right? But the truth is it just may not be. The older your file, the longer you use it, and the increased entropy that results may be putting you on the brink of a major league data crisis.

Angela Meharg Signature & Certifications

 

 

Resources

How to do a Rebuild

Contact us for a list of amazing ProAdvisors who specialize in New File Creation!

Problems Opening QuickBooks

Sometimes things go wrong with QuickBooks – the application, not the company file, at least not in this case. The installation itself can get corrupted, and a big clean up needs to happen. This is especially true when installing new versions. If you’re having a problem opening your company file, and the File Doctor isn’t really helping, try the following steps:

  1. Look at the Services… If there is more than one DB services manager running, stop and disable all but the newest.
  2. Read this article by Intuit about reboot.bat and follow the instructions to run it. https://quickbooks.intuit.com/community/Getting-Started/Run-Reboot-bat/m-p/201698
  3. Where is the company file stored? Move it to a Public Directory. If there are no issues with opening the file, there are permission issues for the QQube network user for the original file location.
  4. Turn off anti-virus temporarily, especially during reinstall.
  5. Confirm that there are no QuickBooks/Internet Explorer security issues.
  6. If UAC was turned to lowest settings during QB installation, uninstall and reinstall with Recommended Settings.
If these things don’t help, you really need to escalate a case with Intuit support.

How to Perform a QuickBooks Rebuild

There are four reasons you may need to perform a reload of QQube:

  1. There is a discrepancy between the data in QuickBooks and your custom report
  2. You have merged two accounts in the QuickBooks Chart of Accounts
  3. You have performed a Rebuild in QuickBooks
  4. You have upgraded QuickBooks to a new year’s version (e.g. 2019 to 2020)

Reason 1 is the most common. If we have determined that your QuickBooks data needs to go through a Rebuild process, please follow these steps. All of this can and should be done by a technical person on your team. The instructions are as follows:

  1. Perform a Verify and Rebuild of your QuickBooks Company File(s).
    1. This should be performed ON THE SERVER or HOSTED MACHINE where the QB file is stored. You will need to be in Single User Mode.
    2. Run a Verify first – but note this doesn’t do ANYTHING to fix your file
    3. IMPORTANT for versions prior to 2018 – if you see ANY reference to an error with “MasterKey encryption” – STOP EVERYTHING and contact QuickBooks support and ask for Intuit Data Services. Proceeding with a Rebuild will likely make your file so corrupt as to be unrecoverable.
    4. Ideally you will perform this first on a restored backup copy as a test. Here are some helpful guidelines, published by a trusted QuickBooks expert in our network: https://blog.quickbooksusers.com/quickbooks/how-to-rebuild-a-quickbooks-file/
  2. Perform a “Portable Copy Round Trip”
    1. This step may not be required, but it’s good housekeeping. Especially if you have lots of DB File Fragments. (Open your file, hit F2 on your keyboard to open the Product Info Window – you’ll see how many DB File Fragments are listed. If more than 9, please proceed with this step.)
    2. After rebuilding enough times to get zero errors, follow these steps to both create and then restore the Portable Copy. (Rename your current file by adding the word “Archive” to the file name so you don’t lose it.) This process reindexes the data, reduces database file fragments, and helps with stability.
    3. https://quickbooks.intuit.com/community/Getting-Started/Create-or-restore-a-QuickBooks-portable-company-file/m-p/203400
  3. Reload the QQube. You have two options: (these features are only available on QQube 6.X and higher)
    1. OK to run this during the work day? (e.g. your file isn’t that big and there isn’t a lot of activity happening in QB that day) 
      • Open the QQube Configuration Tool and go to Manual Refresh. Do this on the Server or PC the QQube Server component is installed. On a Remote Desktop, log in as the QQube_Synch User. Perform a manual refresh after checking the boxes noted here:
    2. Let the server do it overnight
      •  Open the QQube Configuration Tool and go to Scheduler. Check the box noted here:
        QQube Scheduler
      • ***Set a reminder to go back the next day to UNCHECK that box, or it will perform a clear/reload every night!***

If you have any questions, or want us to perform this task for you on a scheduled basis, please read more about our Report & Support Plans or email support@datisfy.com.

How to Create a QuickBooks Portable Copy

To create QuickBooks Portable Copy First, make sure you are in single user mode. Then go to File > Create Copy…

Create Portable File

Select Portable Company File and click Next

QuickBooks Portable Company File

Give it any name you want and click Save

Portable File Save

You will see the following warning message. Click OK

Portable File Save Confirm

A message appears showing that the Portable File creation is in progress.

Portable File Being Created

When the process is complete, a window appears showing the directory where the file was stored.

Portable File Complete

6 Reasons Dashboards Don’t Work – And What To Do About It

So You Think You Want a Dashboard?

I truly hope not.  That is, I truly hope not if you think it won’t require significant effort on your part.

In case you’ve missed this concept over the past decade, a business intelligence dashboard is a data visualization tool that displays the current status of metrics and key performance indicators (KPIs) for an enterprise. Dashboards consolidate and arrange numbers, metrics and sometimes performance scorecards on a single screen.

What’s the Problem?

The problem is dashboards are surprisingly difficult to create. It’s not because we don’t have the technology. No – we can build them with our eyes closed. Dashboards are difficult to create for the following six reasons, five of which have everything to do with you.

  1. You don’t know what to measure in your enterprise.
    • The point of a dashboard is to present up-to-date insight on meaningful business success metrics. The instant you see the display, you should know exactly what action to take to move the gauge in the desired direction. If you can’t act, your dashboard design has failed.
  2. Every role in your organization needs its own metrics.
    • In any social group (e.g. business), individuals occupy and perform roles. Roles are made up of organized practices; a unit or set of organized reciprocal behaviors or acts.
    • Since each role is different, you’ll need to measure different things – and have different dashboards.
  3. You are measuring the wrong thing.
    • Measuring quotas and results provides you with less insight than you think. Instead, try measuring the actions you have pre-determined SHOULD produce those results.
  4. Your systems don’t – or can’t – capture those actions.
  5. Recognition and remuneration are not tied to taking the right activities – nor to keeping the data current.
    • If your team members aren’t measured on taking actions and updating the data, you have nothing to measure BUT results after the fact. You won’t know what correction to put in.
  6. Thinking and planning are hard work, and you don’t want to do hard work.
    • Even I find it challenging to sit and really focus in these days of distraction. But that’s what it takes to do good dashboard design. You can’t avoid it by hiring a consultant. They will just hound you until you sit down and answer their questions to draw out your goals.

Excellence Requires Effort!

A few years ago I was working on a project for a major US bank. The bank’s team got very excited when they heard that they could have a “dashboard” – a single view of multiple key performance metrics in graphical form. When I asked what they intended to measure – and what decisions and actions they needed to make based on these measures – the responded with “Well, we don’t know. Can’t you just give us the industry standard graphs?”

Well, uh, no.

Do you want to be “industry standard” or do you want excellence?

If you are willing to do some hard thinking, align your recognition practices, and confront your innate energy-conserving laziness, a dashboard might help your business. The tool you select to display your metrics is the least important and final decision in the process. But the endeavor, if done well, could transform your company’s future.

If you’d like to learn more, or schedule a dashboard inquiry session, we’re happy to help. The same questions that we use for custom reporting apply to dashboards.

We get it, At this point in your career, every hour, even every minute you spend wrestling with a spreadsheet is costing your company a pretty penny. Datisfy is here to give you back what you need most - your time, improve your bottom line, and make you and your data look even more competent. Just like we did for these fine folks: