Windows' CRT has a limit of 512 open file descriptors, and fds which are
generated by converting a HANDLE via _get_osfhandle count towards this
limit as well.
Regardless, often you find yourself marshalling back and forth between
native HANDLE objects and fds anyway. If we know from the getgo that
we're going to need to work directly with the handle, we can cut out the
marshalling layer while also not contributing to filling up the CRT's
very limited handle table.
On Unix these functions just delegate directly to the existing set of
functions since an fd *is* the native file type. It would be nice, very
long term, if we could convert most uses of fds to file_t.
Differential Revision: https://reviews.llvm.org/D47688
llvm-svn: 333945
The idea behind WindowsSupport.h is that it's in the source directory so
that windows.h'isms don't leak out into the larger LLVM project. To that
end, any symbol that references a symbol from windows.h must be in this
private header, and not in a public header.
However, we had some useful utility functions in WindowsSupport.h which
have no dependency on the Windows API, but still only make sense on
Windows. Those functions should be usable outside of Support since there
is no risk of causing a windows.h leak. Although this introduces some
preprocessor logic in some header files, It's not too egregious and it's
better than the alternative of duplicating a ton of code.
Differential Revision: https://reviews.llvm.org/D47662
llvm-svn: 333798
The return value of sys::getDefaultTargetTriple, which is derived from
-DLLVM_DEFAULT_TRIPLE, is used to construct tool names, default target,
and in the future also to control the search path directly; as such it
should be used textually, without interpretation by LLVM.
Normalization of this value may lead to unexpected results, for example
if we configure LLVM with -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-linux-gnu,
normalization will transform that value to x86_64--linux-gnu. Driver will
use that value to search for tools prefixed with x86_64--linux-gnu- which
may be confusing. This is also inconsistent with the behavior of the
--target flag which is taken as-is without any normalization and overrides
the value of LLVM_DEFAULT_TARGET_TRIPLE.
Users of sys::getDefaultTargetTriple already perform their own
normalization as needed, so this change shouldn't impact existing logic.
Differential Revision: https://reviews.llvm.org/D47153
llvm-svn: 333307
The return value of sys::getDefaultTargetTriple, which is derived from
-DLLVM_DEFAULT_TRIPLE, is used to construct tool names, default target,
and in the future also to control the search path directly; as such it
should be used textually, without interpretation by LLVM.
Normalization of this value may lead to unexpected results, for example
if we configure LLVM with -DLLVM_DEFAULT_TARGET_TRIPLE=x86_64-linux-gnu,
normalization will transform that value to x86_64--linux-gnu. Driver will
use that value to search for tools prefixed with x86_64--linux-gnu- which
may be confusing. This is also inconsistent with the behavior of the
--target flag which is taken as-is without any normalization and overrides
the value of LLVM_DEFAULT_TARGET_TRIPLE.
Users of sys::getDefaultTargetTriple already perform their own
normalization as needed, so this change shouldn't impact existing logic.
Differential Revision: https://reviews.llvm.org/D46910
llvm-svn: 332750
Summary:
Before this patch, signal handling wasn't signal safe. This leads to real-world
crashes. It used ManagedStatic inside of signals, this can allocate and can lead
to unexpected state when a signal occurs during llvm_shutdown (because
llvm_shutdown destroys the ManagedStatic). It also used cl::opt without custom
backing storage. Some de-allocation was performed as well. Acquiring a lock in a
signal handler is also a great way to deadlock.
We can't just disable signals on llvm_shutdown because the signals might do
useful work during that shutdown. We also can't just disable llvm_shutdown for
programs (instead of library uses of clang) because we'd have to then mark the
pointers as not leaked and make sure all the ManagedStatic uses are OK to leak
and remain so.
Move all of the code to lock-free datastructures instead, and avoid having any
of them in an inconsistent state. I'm not trying to be fancy, I'm not using any
explicit memory order because this code isn't hot. The only purpose of the
atomics is to guarantee that a signal firing on the same or a different thread
doesn't see an inconsistent state and crash. In some cases we might miss some
state (for example, we might fail to delete a temporary file), but that's fine.
Note that I haven't touched any of the backtrace support despite it not
technically being totally signal-safe. When that code is called we know
something bad is up and we don't expect to continue execution, so calling
something that e.g. sets errno is the least of our problems.
A similar patch should be applied to lib/Support/Windows/Signals.inc, but that
can be done separately.
Fix r332428 which I reverted in r332429. I originally used double-wide CAS
because I was lazy, but some platforms use a runtime function for that which
thankfully failed to link (it would have been bad for signal handlers
otherwise). I use a separate flag to guard the data instead.
<rdar://problem/28010281>
Reviewers: dexonsmith
Subscribers: steven_wu, llvm-commits
llvm-svn: 332496
Summary:
Before this patch, signal handling wasn't signal safe. This leads to real-world
crashes. It used ManagedStatic inside of signals, this can allocate and can lead
to unexpected state when a signal occurs during llvm_shutdown (because
llvm_shutdown destroys the ManagedStatic). It also used cl::opt without custom
backing storage. Some de-allocation was performed as well. Acquiring a lock in a
signal handler is also a great way to deadlock.
We can't just disable signals on llvm_shutdown because the signals might do
useful work during that shutdown. We also can't just disable llvm_shutdown for
programs (instead of library uses of clang) because we'd have to then mark the
pointers as not leaked and make sure all the ManagedStatic uses are OK to leak
and remain so.
Move all of the code to lock-free datastructures instead, and avoid having any
of them in an inconsistent state. I'm not trying to be fancy, I'm not using any
explicit memory order because this code isn't hot. The only purpose of the
atomics is to guarantee that a signal firing on the same or a different thread
doesn't see an inconsistent state and crash. In some cases we might miss some
state (for example, we might fail to delete a temporary file), but that's fine.
Note that I haven't touched any of the backtrace support despite it not
technically being totally signal-safe. When that code is called we know
something bad is up and we don't expect to continue execution, so calling
something that e.g. sets errno is the least of our problems.
A similar patch should be applied to lib/Support/Windows/Signals.inc, but that
can be done separately.
<rdar://problem/28010281>
Reviewers: dexonsmith
Subscribers: aheejin, llvm-commits
Differential Revision: https://reviews.llvm.org/D46858
llvm-svn: 332428
Summary:
Add documentation for the LLVM Support functions `openFileForWrite` and
`openFileForRead`. The `openFileForRead` parameter `RealPath`, in
particular, I think warranted some explanation.
In addition, make the behavior of the functions more consistent across
platforms. Prior to this patch, Windows would set or not set the result
file descriptor based on the nature of the error, whereas Unix would
consistently set it to `-1` if the open failed. Make Windows
consistently set it to `-1` as well.
Test Plan:
1. `ninja check-llvm`
2. `ninja docs-llvm-html`
Reviewers: zturner, rnk, danielmartin, scanon
Reviewed By: danielmartin, scanon
Subscribers: scanon, danielmartin, llvm-commits
Differential Revision: https://reviews.llvm.org/D46499
llvm-svn: 332075
We've been running doxygen with the autobrief option for a couple of
years now. This makes the \brief markers into our comments
redundant. Since they are a visual distraction and we don't want to
encourage more \brief markers in new code either, this patch removes
them all.
Patch produced by
for i in $(git grep -l '\\brief'); do perl -pi -e 's/\\brief //g' $i & done
Differential Revision: https://reviews.llvm.org/D46290
llvm-svn: 331272
Path.inc/widenPath tries to decode the path using both UTF-8 and the default Windows code page.
This is no longer necessary with the new InitLLVM method which ensures that the command line
arguemnts are already UTF-8 on Windows.
llvm-svn: 330266
GetArgumentVector (or GetCommandLineArguments) is very Windows-specific.
I think it doesn't make much sense to provide that function from sys::Process.
I also made a change so that the function takes a BumpPtrAllocator
instead of a SpecificBumpPtrAllocator. The latter is the class to call
dtors, but since char * is trivially destructible, we should use the
former class.
Differential Revision: https://reviews.llvm.org/D45641
llvm-svn: 330216
The commit in SVN r310001 that added support for this actually didn't
use the right struct field for the frame pointer - for ARM, there is
no register named Fp in the CONTEXT struct. On Windows, the R11
register is used as frame pointer.
Differential Revision: https://reviews.llvm.org/D45590
llvm-svn: 329991
Llvm-mc (and tools that use Path.inc on Windows) assume that strings are utf-8
encoded, however, this is not always the case. On Windows the default codepage
is not utf-8, so most of the time the strings are not utf-8 encoded.
The lld test 'format-binary-non-ascii' uses llvm-mc with a file with non-ascii
characters in the name which is how this bug was found. The test fails when run
using Python 3 because it uses properly encoded unicode strings (Python 2 actually
ends up using a byte string which is not utf-8 encoded, so the test passes, but
that's separate issue).
Patch by Stella Stamenova!
llvm-svn: 329468
These used to be set in the old autoconf build, but the cmake build has had a
"TODO: actually check for these" comment since it was checked in, and they
were set to 1 on mingw unconditionally. It seems safe to say that they always
exist under mingw, so just remove them and assume they're set exactly when on
mingw (with msvc, we use `pragma comment` instead of linking these via flags).
llvm-svn: 328992
This is like MemoryBuffer (read-only) and WritableMemoryBuffer
(writable private), but where the underlying file can be modified
after writing. This is useful when you want to open a file, make
some targeted edits, and then write it back out.
Differential Revision: https://reviews.llvm.org/D44230
llvm-svn: 327057
This is the second part of recommit of r325224. The previous part was
committed in r325426, which deals with C++ memory allocation. Solution
for C memory allocation involved functions `llvm::malloc` and similar.
This was a fragile solution because it caused ambiguity errors in some
cases. In this commit the new functions have names like `llvm::safe_malloc`.
The relevant part of original comment is below, updated for new function
names.
Analysis of fails in the case of out of memory errors can be tricky on
Windows. Such error emerges at the point where memory allocation function
fails, but manifests itself when null pointer is used. These two points
may be distant from each other. Besides, next runs may not exhibit
allocation error.
In some cases memory is allocated by a call to some of C allocation
functions, malloc, calloc and realloc. They are used for interoperability
with C code, when allocated object has variable size and when it is
necessary to avoid call of constructors. In many calls the result is not
checked for null pointer. To simplify checks, new functions are defined
in the namespace 'llvm': `safe_malloc`, `safe_calloc` and `safe_realloc`.
They behave as corresponding standard functions but produce fatal error if
allocation fails. This change replaces the standard functions like 'malloc'
in the cases when the result of the allocation function is not checked
for null pointer.
Finally, there are plain C code, that uses malloc and similar functions. If
the result is not checked, assert statement is added.
Differential Revision: https://reviews.llvm.org/D43010
llvm-svn: 325551
There is a latent Windows kernel bug, the exact trigger
conditions are not well understood, which can cause a file
to be correctly written, but unable to be correctly read.
The workaround appears to be simply calling FlushFileBuffers.
Differential Revision: https://reviews.llvm.org/D42925
llvm-svn: 325274
Analysis of fails in the case of out of memory errors can be tricky on
Windows. Such error emerges at the point where memory allocation function
fails, but manifests itself when null pointer is used. These two points
may be distant from each other. Besides, next runs may not exhibit
allocation error.
Usual programming practice does not require checking result of 'operator
new' because it throws 'std::bad_alloc' in the case of allocation error.
However, LLVM is usually built with exceptions turned off, so 'new' can
return null pointer. This change installs custom new handler, which causes
fatal error in the case of out of memory. The handler is installed
automatically prior to call to 'main' during construction of a static
object defined in 'lib/Support/ErrorHandling.cpp'. If the application does
not use this file, the handler may be installed manually by a call to
'llvm::install_out_of_memory_new_handler', declared in
'include/llvm/Support/ErrorHandling.h".
There are calls to C allocation functions, malloc, calloc and realloc.
They are used for interoperability with C code, when allocated object has
variable size and when it is necessary to avoid call of constructors. In
many calls the result is not checked against null pointer. To simplify
checks, new functions are defined in the namespace 'llvm' with the
same names as these C function. These functions produce fatal error if
allocation fails. User should use 'llvm::malloc' instead of 'std::malloc'
in order to use the safe variant. This change replaces 'std::malloc'
in the cases when the result of allocation function is not checked against
null pointer.
Finally, there are plain C code, that uses malloc and similar functions. If
the result is not checked, assert statements are added.
Differential Revision: https://reviews.llvm.org/D43010
llvm-svn: 325224
Without this when lld failed to replace the output file it would leave
the temporary behind. The problem is that the existing logic is
- cancel the delete flag
- rename
We have to cancel first to avoid renaming and then crashing and
deleting the old version. What is missing then is deleting the
temporary file if the rename fails.
This can be an issue on both unix and windows, but I am not sure how
to cause the rename to fail reliably on unix. I think it can be done
on ZFS since it has an ACL system similar to what windows uses, but
adding support for checking that in llvm-lit is probably not worth it.
llvm-svn: 319786
Removes AllocateRWX, setWritable and setExecutable from sys::Memory and
standardizes on allocateMappedMemory / protectMappedMemory. The
allocateMappedMemory method is updated to request full permissions for memory
blocks so that they can be marked executable later.
llvm-svn: 318464
Summary:
zturner suggested that mapped_file_region::init() on Windows seems to
create mappings that are larger than they need to be: Offset+Size
instead of Size. Indeed, that appears to be the case. I confirmed that
tests pass with mappings of just Size bytes, and fail with Size-1
bytes, suggesting that Size is indeed the correct value.
Reviewers: amccarth, zturner
Reviewed By: zturner
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D39876
llvm-svn: 317850
Summary:
The removed code checks that we are able to handle a 64-bit number, but
the code we're calling takes two dwords (for a total of 64 bits), so this
is always true.
Reviewers: zturner, rnk, majnemer, compnerd
Reviewed By: zturner
Subscribers: amccarth, hiraditya, lebedev.ri, llvm-commits
Differential Revision: https://reviews.llvm.org/D39263
llvm-svn: 316814
In r315079 I added a check for the ERROR_CALL_NOT_IMPLEMENTED error
code, but it turns out earlier versions of Wine just returned false
without setting any error code.
This patch handles the unset error code case.
llvm-svn: 315597
In r315079, fs::rename was reimplemented in terms of CreateFile and
SetFileInformationByHandle. Unfortunately, the latter isn't supported by
Wine. This adds a fallback to MoveFileEx for that case.
Differential Revision: https://reviews.llvm.org/D38817
llvm-svn: 315520
This allows clients to avoid an unnecessary fs::status() call on each
directory entry. Because the information returned by FindFirstFileEx
is a subset of the information returned by a regular status() call,
I needed to extract a base class from file_status that contains only
that information.
On my machine, this reduces the time required to enumerate a ThinLTO
cache directory containing 520k files from almost 4 minutes to less
than 2 seconds.
Differential Revision: https://reviews.llvm.org/D38716
llvm-svn: 315378
Microsoft's debug implementation of std::copy checks if the destination is an
array and then does some bounds checking. This was causing an assertion
failure in fs::rename_internal which copies to a buffer of the appropriate
size but that's type-punned to an array of length 1 for API compatibility
reasons.
Fix is to make make the destination a pointer rather than an array.
llvm-svn: 315222
The current implementation of rename uses ReplaceFile if the
destination file already exists. According to the documentation for
ReplaceFile, the source file is opened without a sharing mode. This
means that there is a short interval of time between when ReplaceFile
renames the file and when it closes the file during which the
destination file cannot be opened.
This behaviour is not POSIX compliant because rename is supposed
to be atomic. It was also causing intermittent link failures when
linking with a ThinLTO cache; the ThinLTO cache implementation expects
all cache files to be openable.
This patch addresses that problem by re-implementing rename
using CreateFile and SetFileInformationByHandle. It is roughly a
reimplementation of ReplaceFile with a better sharing policy as well
as support for renaming in the case where the destination file does
not exist.
This implementation is still not fully POSIX. Specifically in the case
where the destination file is open at the point when rename is called,
there will be a short interval of time during which the destination
file will not exist. It isn't clear whether it is possible to avoid
this using the Windows API.
Differential Revision: https://reviews.llvm.org/D38570
llvm-svn: 315079
Summary:
Change the type of the Redirects parameter of llvm::sys::ExecuteAndWait,
ExecuteNoWait and other APIs that wrap them from `const StringRef **` to
`ArrayRef<Optional<StringRef>>`, which is safer and simplifies the use of these
APIs (no more local StringRef variables just to get a pointer to).
Corresponding clang changes will be posted as a separate patch.
Reviewers: bkramer
Reviewed By: bkramer
Subscribers: vsk, llvm-commits
Differential Revision: https://reviews.llvm.org/D37563
llvm-svn: 313155
Summary:
The function widenPath() for Windows also normalizes long path names by
iterating over the path's components and calling append(). The
assumption during the iteration that separators are not returned by the
iterator doesn't hold because the iterators do return a separator when
the path has a drive name. Handle this case by ignoring separators
during iteration.
Reviewers: rnk
Subscribers: danalbert, srhines
Differential Revision: https://reviews.llvm.org/D36752
llvm-svn: 311382
An environment variable can be in one of three states:
1. undefined.
2. defined with a non-empty value.
3. defined but with an empty value.
The windows implementation did not support case 3
(it was not handling errors). The Linux implementation
is already correct.
Differential Revision: https://reviews.llvm.org/D36394
llvm-svn: 311174
Summary:
Tools like clang that use RemoveFileOnSignal on their output files
weren't actually able to clean up their outputs before this change. Now
the call to llvm::sys::fs::remove succeeds and the temporary file is
deleted. This is a stop-gap to fix clang before implementing the
solution outlined in PR34070.
Reviewers: davide
Subscribers: llvm-commits, hiraditya
Differential Revision: https://reviews.llvm.org/D36337
llvm-svn: 310137
Summary:
It was added to support clang warnings about includes with case
mismatches, but it ended up not being necessary.
Reviewers: twoh, rafael
Subscribers: hiraditya, llvm-commits
Differential Revision: https://reviews.llvm.org/D36328
llvm-svn: 310078
Summary: Different JITs and other clients of LLVM may have different needs in how symbol resolution should occur.
Reviewers: v.g.vassilev, lhames, karies
Reviewed By: v.g.vassilev
Subscribers: pcanal, llvm-commits
Differential Revision: https://reviews.llvm.org/D33529
llvm-svn: 307849
