How to set up/scan C & C++ project via command line? #22091
Replies: 11 comments 12 replies
-
|
was correct. If only 2 files were scanned, then we detected only 2 C/C++ compiler invocations during database creation. The most common reason for this is that build artifacts were not cleaned before creating the database, and subsequently build steps were skipped. Another common reason is the use of caching, e.g., with ccache, which is not supported. It is hard to tell though what is going on without further details. |
Beta Was this translation helpful? Give feedback.
-
|
That didn't help. I deleted all the .obj files in the directory where the compile command is run, and I can see the compilation of every file happening. For each source file I'm seeing: [DATE/TIME] [build-stdout] FILENAME.c The parameters to the cl invocation are: cl /I "path\to\supportclasses" /W3 /MT /GS /RTC1 /bigobj /EHsc /nologo /MP /c main.cpp STANDARD_PREFIX*.c STANDARD_PREFIX*.cpp (Note: /I is slash capital I, not L) An .obj file is created for each source file during this process. I'm using 'cl' as the compiler if that makes any difference. A single invocation of cl with wildcards for all the files to compile is used. i.e., STANDARD_PREFIX*.c STANDARD_PREFIX*.cpp along with paths to the supporting classes in other paths. |
Beta Was this translation helpful? Give feedback.
-
|
I think you hit a limitation in our tooling: we don't handle wildcard expansion on Windows. You should be able to confirm this by looking at the |
Beta Was this translation helpful? Give feedback.
-
|
Yes, I'm seeing those errors in the logs. Do you have an open bug report and are working on a fix? If so, any ETA for the fix? In the meantime, is there a workaround on Windows that does work? I.e., can I use a bash shell, or Windows Subsystem for Linux shell, or some other shell that does work? If you have no other suggestions, then I guess I can try it on Rocky 9 instead (or some Linux flavor)? |
Beta Was this translation helpful? Give feedback.
-
|
We have an internal issue open, but fixing this has not been prioritized, because this is pretty much an edge case (build systems generally specify all files explicitly). The workaround would be to specify all files explicitly. I would assume you can do this by using a Bash shell on Windows, but I have not tested that. |
Beta Was this translation helpful? Give feedback.
-
|
Fixed in CodeQL 2.26.1. |
Beta Was this translation helpful? Give feedback.
-
|
The problem with that is that CL, which is the Windows compiler installed and made accessible in the Visual Studio Command prompt when you install Visual studio is NOT accessible via bash or WSL, so you can't run CL outside of a Visual Studio Command prompt. So, wildcard expansion in Windows shell really should be fixed to make this work. |
Beta Was this translation helpful? Give feedback.
-
That's not true. You can make the compiler accessible from any prompt. It's not tied to the command prompts that Visual Studio provides. I can understand that you don't want to go that route that, but then the question is: why can't specify all files explicitly? |
Beta Was this translation helpful? Give feedback.
-
|
It's actually very complicated to make cl work in a non-VS command prompt as you have to run vcvarsall.bat to set up the proper environment for cl AND cl.exe in the same command line. So, while it is technically possible, it is very complicated/hard to get work. And when you have 1500+ source files in each folder, and I have over one hundred folders, listing each source file is a pain. So, it would be massively simpler for the user if you could get wildcard expansion to work in Windows. I'm trying to get the vcvarsall.bat plus cl.exe setup to work but no luck. Way too complicated. |
Beta Was this translation helpful? Give feedback.
-
|
I'm just trying to offer workarounds here, as this is not something we can fix instantaneously. Especially since this is a bit of an edge case from our perspective, as I mentioned already.
Why list them by hand? The CL supports response files, so you could just write a script that generates the appropriate response files. |
Beta Was this translation helpful? Give feedback.
-
|
OK. I tried that but ran into an error. I created two .rsp files, the 1st is the standard parameters to CL, and the 2nd is the list of all the source files to compile. So, I can run this just fine: cl @standardParams.rsp @listOfAllSourceFilesToCompile.rsp listOfAllSourceFilesToCompile.rsp lists1776 source files, 1 per line. This works no problem, but when I pass that cl command as the --command parameter value to: 'codeql database create', it responds with the error: "(eventual cause: IOException "CreateProcess error=206, The filename or extension is too long")" So, it doesn't look like you can compile all of the source files in a single invocation when creating the database. When you invoke codeql database create, the is shows this output: Initializing database at: path\to\DBNAME |
Beta Was this translation helpful? Give feedback.
-
|
I will need a reproducing test case if you want us to look into that further. From this I also suspect that fixing wildcard expansion will not be sufficient in you case; with wildcard expansion fixed, I expect we will see this error too. |
Beta Was this translation helpful? Give feedback.
-
|
You're hitting the response file expansion from the CodeQL instead of the one from |
Beta Was this translation helpful? Give feedback.
-
|
You can simply create a project with a source file to compile and duplicate that file 1700 times with different source file names so you can reproduce the same thing. Is there no way to expand the size of the command line to accommodate a much longer command string? Or do things in a different way so it doesn't have to expand it all into a single line, which runs into this line size error? |
Beta Was this translation helpful? Give feedback.
-
|
No, because I think we're running into Windows limitation with regard to size of what can be passed to
Either you should be able to share the code, because you work on an open source project, or you have a GitHub advanced security license and should be able to get me a reproducing test case through a support ticket. |
Beta Was this translation helpful? Give feedback.
-
|
My project is not in GitHub and it's not shareable. I'm doing this all at the command line locally using a local/private source code repo. |
Beta Was this translation helpful? Give feedback.
-
|
That means you have a GHAS license, and can open a support ticket with some kind of reproducer. I would not like to second guess this problem based on a somewhat vague description you're providing. |
Beta Was this translation helpful? Give feedback.
-
|
As an alternative, can you run the database create a bunch of times, once per source file and build up the database that way so you don't try to compile everything all at once? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, but then you need to use |
Beta Was this translation helpful? Give feedback.
-
|
My build script is a single invocation of CL. So, using the above commands, how would I would I 'build up' the database with multiple invocations? Can you give me an example of compiling 2 files and I assume the way you do the 2nd compile is just repeated for each of the rest of the files, and then is there a command at the end to finalize the database, or whatever, that you need to do before your run the analyze step? |
Beta Was this translation helpful? Give feedback.
-
|
The easiest way is to create a script, e.g. a batch script, with multiple cl invocations. Then do what you did before. The hard way would be something like the following, but I'm not sure why you would make it hard on yourself by calling |
Beta Was this translation helpful? Give feedback.
-
|
OK. The hard way seems to work. With that said, it would still be very helpful to get wildcarding to work on Windows. Even if the project was too big to compile all at once I could break it up into chunks, so I don't have to call the database trace-command on every individual source file. That would make the building of the database process MUCH faster. Can you explain the 'easy' way? I.e., can you pass a batch file to a single database trace-command and it monitors the CL compile for each individual file being compiled by the batch file? e.g., something like this: codeql database trace-command path/to/db -- compileAllForThisFolder.bat |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
I started with the following:
And when the analyze completes, it says: "CodeQL scanned 2 out of 1562 C/C++ files in this invocation. Typically CodeQL is configured to analyze a single CodeQL language per invocation, so check other invocations to determine overall coverage information."
I looked at the documentation, and apparently you can set the language to multiple languages, e.g., --language=cpp,c
But when I do that, it returned an error during database create that I had to also specify --db-cluster. So, I added that, so now my first command is:
But then, when I run the database analyze step, I now get this error:
"A fatal error has occurred: path\to\DBNAME is not a recognized CodeQL database."
How do I fix that?
Beta Was this translation helpful? Give feedback.
All reactions