rustc_metadata/locator.rs
1//! Finds crate binaries and loads their metadata
2//!
3//! Might I be the first to welcome you to a world of platform differences,
4//! version requirements, dependency graphs, conflicting desires, and fun! This
5//! is the major guts (along with metadata::creader) of the compiler for loading
6//! crates and resolving dependencies. Let's take a tour!
7//!
8//! # The problem
9//!
10//! Each invocation of the compiler is immediately concerned with one primary
11//! problem, to connect a set of crates to resolved crates on the filesystem.
12//! Concretely speaking, the compiler follows roughly these steps to get here:
13//!
14//! 1. Discover a set of `extern crate` statements.
15//! 2. Transform these directives into crate names. If the directive does not
16//! have an explicit name, then the identifier is the name.
17//! 3. For each of these crate names, find a corresponding crate on the
18//! filesystem.
19//!
20//! Sounds easy, right? Let's walk into some of the nuances.
21//!
22//! ## Transitive Dependencies
23//!
24//! Let's say we've got three crates: A, B, and C. A depends on B, and B depends
25//! on C. When we're compiling A, we primarily need to find and locate B, but we
26//! also end up needing to find and locate C as well.
27//!
28//! The reason for this is that any of B's types could be composed of C's types,
29//! any function in B could return a type from C, etc. To be able to guarantee
30//! that we can always type-check/translate any function, we have to have
31//! complete knowledge of the whole ecosystem, not just our immediate
32//! dependencies.
33//!
34//! So now as part of the "find a corresponding crate on the filesystem" step
35//! above, this involves also finding all crates for *all upstream
36//! dependencies*. This includes all dependencies transitively.
37//!
38//! ## Rlibs and Dylibs
39//!
40//! The compiler has two forms of intermediate dependencies. These are dubbed
41//! rlibs and dylibs for the static and dynamic variants, respectively. An rlib
42//! is a rustc-defined file format (currently just an ar archive) while a dylib
43//! is a platform-defined dynamic library. Each library has a metadata somewhere
44//! inside of it.
45//!
46//! A third kind of dependency is an rmeta file. These are metadata files and do
47//! not contain any code, etc. To a first approximation, these are treated in the
48//! same way as rlibs. Where there is both an rlib and an rmeta file, the rlib
49//! gets priority (even if the rmeta file is newer). An rmeta file is only
50//! useful for checking a downstream crate, attempting to link one will cause an
51//! error.
52//!
53//! When translating a crate name to a crate on the filesystem, we all of a
54//! sudden need to take into account both rlibs and dylibs! Linkage later on may
55//! use either one of these files, as each has their pros/cons. The job of crate
56//! loading is to discover what's possible by finding all candidates.
57//!
58//! Most parts of this loading systems keep the dylib/rlib as just separate
59//! variables.
60//!
61//! ## Where to look?
62//!
63//! We can't exactly scan your whole hard drive when looking for dependencies,
64//! so we need to places to look. Currently the compiler will implicitly add the
65//! target lib search path ($prefix/lib/rustlib/$target/lib) to any compilation,
66//! and otherwise all -L flags are added to the search paths.
67//!
68//! ## What criterion to select on?
69//!
70//! This is a pretty tricky area of loading crates. Given a file, how do we know
71//! whether it's the right crate? Currently, the rules look along these lines:
72//!
73//! 1. Does the filename match an rlib/dylib pattern? That is to say, does the
74//! filename have the right prefix/suffix?
75//! 2. Does the filename have the right prefix for the crate name being queried?
76//! This is filtering for files like `libfoo*.rlib` and such. If the crate
77//! we're looking for was originally compiled with -C extra-filename, the
78//! extra filename will be included in this prefix to reduce reading
79//! metadata from crates that would otherwise share our prefix.
80//! 3. Is the file an actual rust library? This is done by loading the metadata
81//! from the library and making sure it's actually there.
82//! 4. Does the name in the metadata agree with the name of the library?
83//! 5. Does the target in the metadata agree with the current target?
84//! 6. Does the SVH match? (more on this later)
85//!
86//! If the file answers `yes` to all these questions, then the file is
87//! considered as being *candidate* for being accepted. It is illegal to have
88//! more than two candidates as the compiler has no method by which to resolve
89//! this conflict. Additionally, rlib/dylib candidates are considered
90//! separately.
91//!
92//! After all this has happened, we have 1 or two files as candidates. These
93//! represent the rlib/dylib file found for a library, and they're returned as
94//! being found.
95//!
96//! ### What about versions?
97//!
98//! A lot of effort has been put forth to remove versioning from the compiler.
99//! There have been forays in the past to have versioning baked in, but it was
100//! largely always deemed insufficient to the point that it was recognized that
101//! it's probably something the compiler shouldn't do anyway due to its
102//! complicated nature and the state of the half-baked solutions.
103//!
104//! With a departure from versioning, the primary criterion for loading crates
105//! is just the name of a crate. If we stopped here, it would imply that you
106//! could never link two crates of the same name from different sources
107//! together, which is clearly a bad state to be in.
108//!
109//! To resolve this problem, we come to the next section!
110//!
111//! # Expert Mode
112//!
113//! A number of flags have been added to the compiler to solve the "version
114//! problem" in the previous section, as well as generally enabling more
115//! powerful usage of the crate loading system of the compiler. The goal of
116//! these flags and options are to enable third-party tools to drive the
117//! compiler with prior knowledge about how the world should look.
118//!
119//! ## The `--extern` flag
120//!
121//! The compiler accepts a flag of this form a number of times:
122//!
123//! ```text
124//! --extern crate-name=path/to/the/crate.rlib
125//! ```
126//!
127//! This flag is basically the following letter to the compiler:
128//!
129//! > Dear rustc,
130//! >
131//! > When you are attempting to load the immediate dependency `crate-name`, I
132//! > would like you to assume that the library is located at
133//! > `path/to/the/crate.rlib`, and look nowhere else. Also, please do not
134//! > assume that the path I specified has the name `crate-name`.
135//!
136//! This flag basically overrides most matching logic except for validating that
137//! the file is indeed a rust library. The same `crate-name` can be specified
138//! twice to specify the rlib/dylib pair.
139//!
140//! ## Enabling "multiple versions"
141//!
142//! This basically boils down to the ability to specify arbitrary packages to
143//! the compiler. For example, if crate A wanted to use Bv1 and Bv2, then it
144//! would look something like:
145//!
146//! ```compile_fail,E0463
147//! extern crate b1;
148//! extern crate b2;
149//!
150//! fn main() {}
151//! ```
152//!
153//! and the compiler would be invoked as:
154//!
155//! ```text
156//! rustc a.rs --extern b1=path/to/libb1.rlib --extern b2=path/to/libb2.rlib
157//! ```
158//!
159//! In this scenario there are two crates named `b` and the compiler must be
160//! manually driven to be informed where each crate is.
161//!
162//! ## Frobbing symbols
163//!
164//! One of the immediate problems with linking the same library together twice
165//! in the same problem is dealing with duplicate symbols. The primary way to
166//! deal with this in rustc is to add hashes to the end of each symbol.
167//!
168//! In order to force hashes to change between versions of a library, if
169//! desired, the compiler exposes an option `-C metadata=foo`, which is used to
170//! initially seed each symbol hash. The string `foo` is prepended to each
171//! string-to-hash to ensure that symbols change over time.
172//!
173//! ## Loading transitive dependencies
174//!
175//! Dealing with same-named-but-distinct crates is not just a local problem, but
176//! one that also needs to be dealt with for transitive dependencies. Note that
177//! in the letter above `--extern` flags only apply to the *local* set of
178//! dependencies, not the upstream transitive dependencies. Consider this
179//! dependency graph:
180//!
181//! ```text
182//! A.1 A.2
183//! | |
184//! | |
185//! B C
186//! \ /
187//! \ /
188//! D
189//! ```
190//!
191//! In this scenario, when we compile `D`, we need to be able to distinctly
192//! resolve `A.1` and `A.2`, but an `--extern` flag cannot apply to these
193//! transitive dependencies.
194//!
195//! Note that the key idea here is that `B` and `C` are both *already compiled*.
196//! That is, they have already resolved their dependencies. Due to unrelated
197//! technical reasons, when a library is compiled, it is only compatible with
198//! the *exact same* version of the upstream libraries it was compiled against.
199//! We use the "Strict Version Hash" to identify the exact copy of an upstream
200//! library.
201//!
202//! With this knowledge, we know that `B` and `C` will depend on `A` with
203//! different SVH values, so we crawl the normal `-L` paths looking for
204//! `liba*.rlib` and filter based on the contained SVH.
205//!
206//! In the end, this ends up not needing `--extern` to specify upstream
207//! transitive dependencies.
208//!
209//! # Wrapping up
210//!
211//! That's the general overview of loading crates in the compiler, but it's by
212//! no means all of the necessary details. Take a look at the rest of
213//! metadata::locator or metadata::creader for all the juicy details!
214
215use std::borrow::Cow;
216use std::io::{Result as IoResult, Write};
217use std::ops::Deref;
218use std::path::{Path, PathBuf};
219use std::{cmp, fmt};
220
221use rustc_data_structures::fx::{FxHashSet, FxIndexMap};
222use rustc_data_structures::memmap::Mmap;
223use rustc_data_structures::owned_slice::{OwnedSlice, slice_owned};
224use rustc_data_structures::svh::Svh;
225use rustc_errors::{DiagArgValue, IntoDiagArg};
226use rustc_fs_util::try_canonicalize;
227use rustc_session::cstore::CrateSource;
228use rustc_session::filesearch::FileSearch;
229use rustc_session::search_paths::PathKind;
230use rustc_session::utils::CanonicalizedPath;
231use rustc_session::{Session, config};
232use rustc_span::{Span, Symbol};
233use rustc_target::spec::{Target, TargetTuple};
234use tempfile::Builder as TempFileBuilder;
235use tracing::{debug, info};
236
237use crate::creader::{Library, MetadataLoader};
238use crate::errors;
239use crate::rmeta::{METADATA_HEADER, MetadataBlob, rustc_version};
240
241#[derive(Clone)]
242pub(crate) struct CrateLocator<'a> {
243 // Immutable per-session configuration.
244 only_needs_metadata: bool,
245 sysroot: &'a Path,
246 metadata_loader: &'a dyn MetadataLoader,
247 cfg_version: &'static str,
248
249 // Immutable per-search configuration.
250 crate_name: Symbol,
251 exact_paths: Vec<CanonicalizedPath>,
252 pub hash: Option<Svh>,
253 extra_filename: Option<&'a str>,
254 target: &'a Target,
255 tuple: TargetTuple,
256 filesearch: &'a FileSearch,
257 is_proc_macro: bool,
258 path_kind: PathKind,
259}
260
261#[derive(Clone, Debug)]
262pub(crate) struct CratePaths {
263 pub(crate) name: Symbol,
264 source: CrateSource,
265}
266
267impl CratePaths {
268 pub(crate) fn new(name: Symbol, source: CrateSource) -> CratePaths {
269 CratePaths { name, source }
270 }
271}
272
273#[derive(Copy, Clone, Debug, PartialEq)]
274pub(crate) enum CrateFlavor {
275 Rlib,
276 Rmeta,
277 Dylib,
278 SDylib,
279}
280
281impl fmt::Display for CrateFlavor {
282 fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
283 f.write_str(match *self {
284 CrateFlavor::Rlib => "rlib",
285 CrateFlavor::Rmeta => "rmeta",
286 CrateFlavor::Dylib => "dylib",
287 CrateFlavor::SDylib => "sdylib",
288 })
289 }
290}
291
292impl IntoDiagArg for CrateFlavor {
293 fn into_diag_arg(self, _: &mut Option<std::path::PathBuf>) -> rustc_errors::DiagArgValue {
294 match self {
295 CrateFlavor::Rlib => DiagArgValue::Str(Cow::Borrowed("rlib")),
296 CrateFlavor::Rmeta => DiagArgValue::Str(Cow::Borrowed("rmeta")),
297 CrateFlavor::Dylib => DiagArgValue::Str(Cow::Borrowed("dylib")),
298 CrateFlavor::SDylib => DiagArgValue::Str(Cow::Borrowed("sdylib")),
299 }
300 }
301}
302
303impl<'a> CrateLocator<'a> {
304 pub(crate) fn new(
305 sess: &'a Session,
306 metadata_loader: &'a dyn MetadataLoader,
307 crate_name: Symbol,
308 is_rlib: bool,
309 hash: Option<Svh>,
310 extra_filename: Option<&'a str>,
311 path_kind: PathKind,
312 ) -> CrateLocator<'a> {
313 let needs_object_code = sess.opts.output_types.should_codegen();
314 // If we're producing an rlib, then we don't need object code.
315 // Or, if we're not producing object code, then we don't need it either
316 // (e.g., if we're a cdylib but emitting just metadata).
317 let only_needs_metadata = is_rlib || !needs_object_code;
318
319 CrateLocator {
320 only_needs_metadata,
321 sysroot: sess.opts.sysroot.path(),
322 metadata_loader,
323 cfg_version: sess.cfg_version,
324 crate_name,
325 exact_paths: if hash.is_none() {
326 sess.opts
327 .externs
328 .get(crate_name.as_str())
329 .into_iter()
330 .filter_map(|entry| entry.files())
331 .flatten()
332 .cloned()
333 .collect()
334 } else {
335 // SVH being specified means this is a transitive dependency,
336 // so `--extern` options do not apply.
337 Vec::new()
338 },
339 hash,
340 extra_filename,
341 target: &sess.target,
342 tuple: sess.opts.target_triple.clone(),
343 filesearch: sess.target_filesearch(),
344 path_kind,
345 is_proc_macro: false,
346 }
347 }
348
349 pub(crate) fn for_proc_macro(&mut self, sess: &'a Session, path_kind: PathKind) {
350 self.is_proc_macro = true;
351 self.target = &sess.host;
352 self.tuple = TargetTuple::from_tuple(config::host_tuple());
353 self.filesearch = sess.host_filesearch();
354 self.path_kind = path_kind;
355 }
356
357 pub(crate) fn for_target_proc_macro(&mut self, sess: &'a Session, path_kind: PathKind) {
358 self.is_proc_macro = true;
359 self.target = &sess.target;
360 self.tuple = sess.opts.target_triple.clone();
361 self.filesearch = sess.target_filesearch();
362 self.path_kind = path_kind;
363 }
364
365 pub(crate) fn maybe_load_library_crate(
366 &self,
367 crate_rejections: &mut CrateRejections,
368 ) -> Result<Option<Library>, CrateError> {
369 if !self.exact_paths.is_empty() {
370 return self.find_commandline_library(crate_rejections);
371 }
372 let mut seen_paths = FxHashSet::default();
373 if let Some(extra_filename) = self.extra_filename {
374 if let library @ Some(_) =
375 self.find_library_crate(crate_rejections, extra_filename, &mut seen_paths)?
376 {
377 return Ok(library);
378 }
379 }
380 self.find_library_crate(crate_rejections, "", &mut seen_paths)
381 }
382
383 fn find_library_crate(
384 &self,
385 crate_rejections: &mut CrateRejections,
386 extra_prefix: &str,
387 seen_paths: &mut FxHashSet<PathBuf>,
388 ) -> Result<Option<Library>, CrateError> {
389 let rmeta_prefix = &format!("lib{}{}", self.crate_name, extra_prefix);
390 let rlib_prefix = rmeta_prefix;
391 let dylib_prefix =
392 &format!("{}{}{}", self.target.dll_prefix, self.crate_name, extra_prefix);
393 let staticlib_prefix =
394 &format!("{}{}{}", self.target.staticlib_prefix, self.crate_name, extra_prefix);
395 let interface_prefix = rmeta_prefix;
396
397 let rmeta_suffix = ".rmeta";
398 let rlib_suffix = ".rlib";
399 let dylib_suffix = &self.target.dll_suffix;
400 let staticlib_suffix = &self.target.staticlib_suffix;
401 let interface_suffix = ".rs";
402
403 let mut candidates: FxIndexMap<
404 _,
405 (FxIndexMap<_, _>, FxIndexMap<_, _>, FxIndexMap<_, _>, FxIndexMap<_, _>),
406 > = Default::default();
407
408 // First, find all possible candidate rlibs and dylibs purely based on
409 // the name of the files themselves. We're trying to match against an
410 // exact crate name and a possibly an exact hash.
411 //
412 // During this step, we can filter all found libraries based on the
413 // name and id found in the crate id (we ignore the path portion for
414 // filename matching), as well as the exact hash (if specified). If we
415 // end up having many candidates, we must look at the metadata to
416 // perform exact matches against hashes/crate ids. Note that opening up
417 // the metadata is where we do an exact match against the full contents
418 // of the crate id (path/name/id).
419 //
420 // The goal of this step is to look at as little metadata as possible.
421 // Unfortunately, the prefix-based matching sometimes is over-eager.
422 // E.g. if `rlib_suffix` is `libstd` it'll match the file
423 // `libstd_detect-8d6701fb958915ad.rlib` (incorrect) as well as
424 // `libstd-f3ab5b1dea981f17.rlib` (correct). But this is hard to avoid
425 // given that `extra_filename` comes from the `-C extra-filename`
426 // option and thus can be anything, and the incorrect match will be
427 // handled safely in `extract_one`.
428 for search_path in self.filesearch.search_paths(self.path_kind) {
429 debug!("searching {}", search_path.dir.display());
430 let spf = &search_path.files;
431
432 let mut should_check_staticlibs = true;
433 for (prefix, suffix, kind) in [
434 (rlib_prefix.as_str(), rlib_suffix, CrateFlavor::Rlib),
435 (rmeta_prefix.as_str(), rmeta_suffix, CrateFlavor::Rmeta),
436 (dylib_prefix, dylib_suffix, CrateFlavor::Dylib),
437 (interface_prefix, interface_suffix, CrateFlavor::SDylib),
438 ] {
439 if prefix == staticlib_prefix && suffix == staticlib_suffix {
440 should_check_staticlibs = false;
441 }
442 if let Some(matches) = spf.query(prefix, suffix) {
443 for (hash, spf) in matches {
444 info!("lib candidate: {}", spf.path.display());
445
446 let (rlibs, rmetas, dylibs, interfaces) =
447 candidates.entry(hash).or_default();
448 {
449 // As a performance optimisation we canonicalize the path and skip
450 // ones we've already seen. This allows us to ignore crates
451 // we know are exactual equal to ones we've already found.
452 // Going to the same crate through different symlinks does not change the result.
453 let path = try_canonicalize(&spf.path)
454 .unwrap_or_else(|_| spf.path.to_path_buf());
455 if seen_paths.contains(&path) {
456 continue;
457 };
458 seen_paths.insert(path);
459 }
460 // Use the original path (potentially with unresolved symlinks),
461 // filesystem code should not care, but this is nicer for diagnostics.
462 let path = spf.path.to_path_buf();
463 match kind {
464 CrateFlavor::Rlib => rlibs.insert(path, search_path.kind),
465 CrateFlavor::Rmeta => rmetas.insert(path, search_path.kind),
466 CrateFlavor::Dylib => dylibs.insert(path, search_path.kind),
467 CrateFlavor::SDylib => interfaces.insert(path, search_path.kind),
468 };
469 }
470 }
471 }
472 if let Some(static_matches) = should_check_staticlibs
473 .then(|| spf.query(staticlib_prefix, staticlib_suffix))
474 .flatten()
475 {
476 for (_, spf) in static_matches {
477 crate_rejections.via_kind.push(CrateMismatch {
478 path: spf.path.to_path_buf(),
479 got: "static".to_string(),
480 });
481 }
482 }
483 }
484
485 // We have now collected all known libraries into a set of candidates
486 // keyed of the filename hash listed. For each filename, we also have a
487 // list of rlibs/dylibs that apply. Here, we map each of these lists
488 // (per hash), to a Library candidate for returning.
489 //
490 // A Library candidate is created if the metadata for the set of
491 // libraries corresponds to the crate id and hash criteria that this
492 // search is being performed for.
493 let mut libraries = FxIndexMap::default();
494 for (_hash, (rlibs, rmetas, dylibs, interfaces)) in candidates {
495 if let Some((svh, lib)) =
496 self.extract_lib(crate_rejections, rlibs, rmetas, dylibs, interfaces)?
497 {
498 libraries.insert(svh, lib);
499 }
500 }
501
502 // Having now translated all relevant found hashes into libraries, see
503 // what we've got and figure out if we found multiple candidates for
504 // libraries or not.
505 match libraries.len() {
506 0 => Ok(None),
507 1 => Ok(Some(libraries.into_iter().next().unwrap().1)),
508 _ => {
509 let mut candidates: Vec<PathBuf> = libraries
510 .into_values()
511 .map(|lib| lib.source.paths().next().unwrap().clone())
512 .collect();
513 candidates.sort();
514
515 Err(CrateError::MultipleCandidates(
516 self.crate_name,
517 // these are the same for all candidates
518 get_flavor_from_path(candidates.first().unwrap()),
519 candidates,
520 ))
521 }
522 }
523 }
524
525 fn extract_lib(
526 &self,
527 crate_rejections: &mut CrateRejections,
528 rlibs: FxIndexMap<PathBuf, PathKind>,
529 rmetas: FxIndexMap<PathBuf, PathKind>,
530 dylibs: FxIndexMap<PathBuf, PathKind>,
531 interfaces: FxIndexMap<PathBuf, PathKind>,
532 ) -> Result<Option<(Svh, Library)>, CrateError> {
533 let mut slot = None;
534 // Order here matters, rmeta should come first.
535 //
536 // Make sure there's at most one rlib and at most one dylib.
537 //
538 // See comment in `extract_one` below.
539 let rmeta = self.extract_one(crate_rejections, rmetas, CrateFlavor::Rmeta, &mut slot)?;
540 let rlib = self.extract_one(crate_rejections, rlibs, CrateFlavor::Rlib, &mut slot)?;
541 let sdylib_interface =
542 self.extract_one(crate_rejections, interfaces, CrateFlavor::SDylib, &mut slot)?;
543 let dylib = self.extract_one(crate_rejections, dylibs, CrateFlavor::Dylib, &mut slot)?;
544
545 if sdylib_interface.is_some() && dylib.is_none() {
546 return Err(CrateError::FullMetadataNotFound(self.crate_name, CrateFlavor::SDylib));
547 }
548
549 let source = CrateSource { rmeta, rlib, dylib, sdylib_interface };
550 Ok(slot.map(|(svh, metadata, _, _)| (svh, Library { source, metadata })))
551 }
552
553 fn needs_crate_flavor(&self, flavor: CrateFlavor) -> bool {
554 if flavor == CrateFlavor::Dylib && self.is_proc_macro {
555 return true;
556 }
557
558 if self.only_needs_metadata {
559 flavor == CrateFlavor::Rmeta
560 } else {
561 // we need all flavors (perhaps not true, but what we do for now)
562 true
563 }
564 }
565
566 // Attempts to extract *one* library from the set `m`. If the set has no
567 // elements, `None` is returned. If the set has more than one element, then
568 // the errors and notes are emitted about the set of libraries.
569 //
570 // With only one library in the set, this function will extract it, and then
571 // read the metadata from it if `*slot` is `None`. If the metadata couldn't
572 // be read, it is assumed that the file isn't a valid rust library (no
573 // errors are emitted).
574 //
575 // The `PathBuf` in `slot` will only be used for diagnostic purposes.
576 fn extract_one(
577 &self,
578 crate_rejections: &mut CrateRejections,
579 m: FxIndexMap<PathBuf, PathKind>,
580 flavor: CrateFlavor,
581 slot: &mut Option<(Svh, MetadataBlob, PathBuf, CrateFlavor)>,
582 ) -> Result<Option<(PathBuf, PathKind)>, CrateError> {
583 // If we are producing an rlib, and we've already loaded metadata, then
584 // we should not attempt to discover further crate sources (unless we're
585 // locating a proc macro; exact logic is in needs_crate_flavor). This means
586 // that under -Zbinary-dep-depinfo we will not emit a dependency edge on
587 // the *unused* rlib, and by returning `None` here immediately we
588 // guarantee that we do indeed not use it.
589 //
590 // See also #68149 which provides more detail on why emitting the
591 // dependency on the rlib is a bad thing.
592 if slot.is_some() {
593 if m.is_empty() || !self.needs_crate_flavor(flavor) {
594 return Ok(None);
595 }
596 }
597
598 let mut ret: Option<(PathBuf, PathKind)> = None;
599 let mut err_data: Option<Vec<PathBuf>> = None;
600 for (lib, kind) in m {
601 info!("{} reading metadata from: {}", flavor, lib.display());
602 if flavor == CrateFlavor::Rmeta && lib.metadata().is_ok_and(|m| m.len() == 0) {
603 // Empty files will cause get_metadata_section to fail. Rmeta
604 // files can be empty, for example with binaries (which can
605 // often appear with `cargo check` when checking a library as
606 // a unittest). We don't want to emit a user-visible warning
607 // in this case as it is not a real problem.
608 debug!("skipping empty file");
609 continue;
610 }
611 let (hash, metadata) = match get_metadata_section(
612 self.target,
613 flavor,
614 &lib,
615 self.metadata_loader,
616 self.cfg_version,
617 Some(self.crate_name),
618 ) {
619 Ok(blob) => {
620 if let Some(h) = self.crate_matches(crate_rejections, &blob, &lib) {
621 (h, blob)
622 } else {
623 info!("metadata mismatch");
624 continue;
625 }
626 }
627 Err(MetadataError::VersionMismatch { expected_version, found_version }) => {
628 // The file was present and created by the same compiler version, but we
629 // couldn't load it for some reason. Give a hard error instead of silently
630 // ignoring it, but only if we would have given an error anyway.
631 info!(
632 "Rejecting via version: expected {} got {}",
633 expected_version, found_version
634 );
635 crate_rejections
636 .via_version
637 .push(CrateMismatch { path: lib, got: found_version });
638 continue;
639 }
640 Err(MetadataError::LoadFailure(err)) => {
641 info!("no metadata found: {}", err);
642 // Metadata was loaded from interface file earlier.
643 if let Some((.., CrateFlavor::SDylib)) = slot {
644 ret = Some((lib, kind));
645 continue;
646 }
647 // The file was present and created by the same compiler version, but we
648 // couldn't load it for some reason. Give a hard error instead of silently
649 // ignoring it, but only if we would have given an error anyway.
650 crate_rejections.via_invalid.push(CrateMismatch { path: lib, got: err });
651 continue;
652 }
653 Err(err @ MetadataError::NotPresent(_)) => {
654 info!("no metadata found: {}", err);
655 continue;
656 }
657 };
658 // If we see multiple hashes, emit an error about duplicate candidates.
659 if slot.as_ref().is_some_and(|s| s.0 != hash) {
660 if let Some(candidates) = err_data {
661 return Err(CrateError::MultipleCandidates(
662 self.crate_name,
663 flavor,
664 candidates,
665 ));
666 }
667 err_data = Some(vec![slot.take().unwrap().2]);
668 }
669 if let Some(candidates) = &mut err_data {
670 candidates.push(lib);
671 continue;
672 }
673
674 // Ok so at this point we've determined that `(lib, kind)` above is
675 // a candidate crate to load, and that `slot` is either none (this
676 // is the first crate of its kind) or if some the previous path has
677 // the exact same hash (e.g., it's the exact same crate).
678 //
679 // In principle these two candidate crates are exactly the same so
680 // we can choose either of them to link. As a stupidly gross hack,
681 // however, we favor crate in the sysroot.
682 //
683 // You can find more info in rust-lang/rust#39518 and various linked
684 // issues, but the general gist is that during testing libstd the
685 // compilers has two candidates to choose from: one in the sysroot
686 // and one in the deps folder. These two crates are the exact same
687 // crate but if the compiler chooses the one in the deps folder
688 // it'll cause spurious errors on Windows.
689 //
690 // As a result, we favor the sysroot crate here. Note that the
691 // candidates are all canonicalized, so we canonicalize the sysroot
692 // as well.
693 if let Some((prev, _)) = &ret {
694 let sysroot = self.sysroot;
695 let sysroot = try_canonicalize(sysroot).unwrap_or_else(|_| sysroot.to_path_buf());
696 if prev.starts_with(&sysroot) {
697 continue;
698 }
699 }
700
701 // We error eagerly here. If we're locating a rlib, then in theory the full metadata
702 // could still be in a (later resolved) dylib. In practice, if the rlib and dylib
703 // were produced in a way where one has full metadata and the other hasn't, it would
704 // mean that they were compiled using different compiler flags and probably also have
705 // a different SVH value.
706 if metadata.get_header().is_stub {
707 // `is_stub` should never be true for .rmeta files.
708 assert_ne!(flavor, CrateFlavor::Rmeta);
709
710 // Because rmeta files are resolved before rlib/dylib files, if this is a stub and
711 // we haven't found a slot already, it means that the full metadata is missing.
712 if slot.is_none() {
713 return Err(CrateError::FullMetadataNotFound(self.crate_name, flavor));
714 }
715 } else {
716 *slot = Some((hash, metadata, lib.clone(), flavor));
717 }
718 ret = Some((lib, kind));
719 }
720
721 if let Some(candidates) = err_data {
722 Err(CrateError::MultipleCandidates(self.crate_name, flavor, candidates))
723 } else {
724 Ok(ret)
725 }
726 }
727
728 fn crate_matches(
729 &self,
730 crate_rejections: &mut CrateRejections,
731 metadata: &MetadataBlob,
732 libpath: &Path,
733 ) -> Option<Svh> {
734 let header = metadata.get_header();
735 if header.is_proc_macro_crate != self.is_proc_macro {
736 info!(
737 "Rejecting via proc macro: expected {} got {}",
738 self.is_proc_macro, header.is_proc_macro_crate,
739 );
740 return None;
741 }
742
743 if self.exact_paths.is_empty() && self.crate_name != header.name {
744 info!("Rejecting via crate name");
745 return None;
746 }
747
748 if header.triple != self.tuple {
749 info!("Rejecting via crate triple: expected {} got {}", self.tuple, header.triple);
750 crate_rejections.via_triple.push(CrateMismatch {
751 path: libpath.to_path_buf(),
752 got: header.triple.to_string(),
753 });
754 return None;
755 }
756
757 let hash = header.hash;
758 if let Some(expected_hash) = self.hash {
759 if hash != expected_hash {
760 info!("Rejecting via hash: expected {} got {}", expected_hash, hash);
761 crate_rejections
762 .via_hash
763 .push(CrateMismatch { path: libpath.to_path_buf(), got: hash.to_string() });
764 return None;
765 }
766 }
767
768 Some(hash)
769 }
770
771 fn find_commandline_library(
772 &self,
773 crate_rejections: &mut CrateRejections,
774 ) -> Result<Option<Library>, CrateError> {
775 // First, filter out all libraries that look suspicious. We only accept
776 // files which actually exist that have the correct naming scheme for
777 // rlibs/dylibs.
778 let mut rlibs = FxIndexMap::default();
779 let mut rmetas = FxIndexMap::default();
780 let mut dylibs = FxIndexMap::default();
781 let mut sdylib_interfaces = FxIndexMap::default();
782 for loc in &self.exact_paths {
783 let loc_canon = loc.canonicalized();
784 let loc_orig = loc.original();
785 if !loc_canon.exists() {
786 return Err(CrateError::ExternLocationNotExist(self.crate_name, loc_orig.clone()));
787 }
788 if !loc_orig.is_file() {
789 return Err(CrateError::ExternLocationNotFile(self.crate_name, loc_orig.clone()));
790 }
791 // Note to take care and match against the non-canonicalized name:
792 // some systems save build artifacts into content-addressed stores
793 // that do not preserve extensions, and then link to them using
794 // e.g. symbolic links. If we canonicalize too early, we resolve
795 // the symlink, the file type is lost and we might treat rlibs and
796 // rmetas as dylibs.
797 let Some(file) = loc_orig.file_name().and_then(|s| s.to_str()) else {
798 return Err(CrateError::ExternLocationNotFile(self.crate_name, loc_orig.clone()));
799 };
800 if file.starts_with("lib") {
801 if file.ends_with(".rlib") {
802 rlibs.insert(loc_canon.clone(), PathKind::ExternFlag);
803 continue;
804 }
805 if file.ends_with(".rmeta") {
806 rmetas.insert(loc_canon.clone(), PathKind::ExternFlag);
807 continue;
808 }
809 if file.ends_with(".rs") {
810 sdylib_interfaces.insert(loc_canon.clone(), PathKind::ExternFlag);
811 }
812 }
813 let dll_prefix = self.target.dll_prefix.as_ref();
814 let dll_suffix = self.target.dll_suffix.as_ref();
815 if file.starts_with(dll_prefix) && file.ends_with(dll_suffix) {
816 dylibs.insert(loc_canon.clone(), PathKind::ExternFlag);
817 continue;
818 }
819 crate_rejections
820 .via_filename
821 .push(CrateMismatch { path: loc_orig.clone(), got: String::new() });
822 }
823
824 // Extract the dylib/rlib/rmeta triple.
825 self.extract_lib(crate_rejections, rlibs, rmetas, dylibs, sdylib_interfaces)
826 .map(|opt| opt.map(|(_, lib)| lib))
827 }
828
829 pub(crate) fn into_error(
830 self,
831 crate_rejections: CrateRejections,
832 dep_root: Option<CratePaths>,
833 ) -> CrateError {
834 CrateError::LocatorCombined(Box::new(CombinedLocatorError {
835 crate_name: self.crate_name,
836 dep_root,
837 triple: self.tuple,
838 dll_prefix: self.target.dll_prefix.to_string(),
839 dll_suffix: self.target.dll_suffix.to_string(),
840 crate_rejections,
841 }))
842 }
843}
844
845fn get_metadata_section<'p>(
846 target: &Target,
847 flavor: CrateFlavor,
848 filename: &'p Path,
849 loader: &dyn MetadataLoader,
850 cfg_version: &'static str,
851 crate_name: Option<Symbol>,
852) -> Result<MetadataBlob, MetadataError<'p>> {
853 if !filename.exists() {
854 return Err(MetadataError::NotPresent(filename));
855 }
856 let raw_bytes = match flavor {
857 CrateFlavor::Rlib => {
858 loader.get_rlib_metadata(target, filename).map_err(MetadataError::LoadFailure)?
859 }
860 CrateFlavor::SDylib => {
861 let compiler = std::env::current_exe().map_err(|_err| {
862 MetadataError::LoadFailure(
863 "couldn't obtain current compiler binary when loading sdylib interface"
864 .to_string(),
865 )
866 })?;
867
868 let tmp_path = match TempFileBuilder::new().prefix("rustc").tempdir() {
869 Ok(tmp_path) => tmp_path,
870 Err(error) => {
871 return Err(MetadataError::LoadFailure(format!(
872 "couldn't create a temp dir: {}",
873 error
874 )));
875 }
876 };
877
878 let crate_name = crate_name.unwrap();
879 debug!("compiling {}", filename.display());
880 // FIXME: This will need to be done either within the current compiler session or
881 // as a separate compiler session in the same process.
882 let res = std::process::Command::new(compiler)
883 .arg(&filename)
884 .arg("--emit=metadata")
885 .arg(format!("--crate-name={}", crate_name))
886 .arg(format!("--out-dir={}", tmp_path.path().display()))
887 .arg("-Zbuild-sdylib-interface")
888 .output()
889 .map_err(|err| {
890 MetadataError::LoadFailure(format!("couldn't compile interface: {}", err))
891 })?;
892
893 if !res.status.success() {
894 return Err(MetadataError::LoadFailure(format!(
895 "couldn't compile interface: {}",
896 std::str::from_utf8(&res.stderr).unwrap_or_default()
897 )));
898 }
899
900 // Load interface metadata instead of crate metadata.
901 let interface_metadata_name = format!("lib{}.rmeta", crate_name);
902 let rmeta_file = tmp_path.path().join(interface_metadata_name);
903 debug!("loading interface metadata from {}", rmeta_file.display());
904 let rmeta = get_rmeta_metadata_section(&rmeta_file)?;
905 let _ = std::fs::remove_file(rmeta_file);
906
907 rmeta
908 }
909 CrateFlavor::Dylib => {
910 let buf =
911 loader.get_dylib_metadata(target, filename).map_err(MetadataError::LoadFailure)?;
912 let header_len = METADATA_HEADER.len();
913 // header + u64 length of data
914 let data_start = header_len + 8;
915
916 debug!("checking {} bytes of metadata-version stamp", header_len);
917 let header = &buf[..cmp::min(header_len, buf.len())];
918 if header != METADATA_HEADER {
919 return Err(MetadataError::LoadFailure(format!(
920 "invalid metadata version found: {}",
921 filename.display()
922 )));
923 }
924
925 // Length of the metadata - this allows linkers to pad the section if they want
926 let Ok(len_bytes) =
927 <[u8; 8]>::try_from(&buf[header_len..cmp::min(data_start, buf.len())])
928 else {
929 return Err(MetadataError::LoadFailure(
930 "invalid metadata length found".to_string(),
931 ));
932 };
933 let metadata_len = u64::from_le_bytes(len_bytes) as usize;
934
935 // Header is okay -> inflate the actual metadata
936 buf.slice(|buf| &buf[data_start..(data_start + metadata_len)])
937 }
938 CrateFlavor::Rmeta => get_rmeta_metadata_section(filename)?,
939 };
940 let Ok(blob) = MetadataBlob::new(raw_bytes) else {
941 return Err(MetadataError::LoadFailure(format!(
942 "corrupt metadata encountered in {}",
943 filename.display()
944 )));
945 };
946 match blob.check_compatibility(cfg_version) {
947 Ok(()) => {
948 debug!("metadata blob read okay");
949 Ok(blob)
950 }
951 Err(None) => Err(MetadataError::LoadFailure(format!(
952 "invalid metadata version found: {}",
953 filename.display()
954 ))),
955 Err(Some(found_version)) => {
956 return Err(MetadataError::VersionMismatch {
957 expected_version: rustc_version(cfg_version),
958 found_version,
959 });
960 }
961 }
962}
963
964fn get_rmeta_metadata_section<'a, 'p>(filename: &'p Path) -> Result<OwnedSlice, MetadataError<'a>> {
965 // mmap the file, because only a small fraction of it is read.
966 let file = std::fs::File::open(filename).map_err(|_| {
967 MetadataError::LoadFailure(format!(
968 "failed to open rmeta metadata: '{}'",
969 filename.display()
970 ))
971 })?;
972 let mmap = unsafe { Mmap::map(file) };
973 let mmap = mmap.map_err(|_| {
974 MetadataError::LoadFailure(format!(
975 "failed to mmap rmeta metadata: '{}'",
976 filename.display()
977 ))
978 })?;
979
980 Ok(slice_owned(mmap, Deref::deref))
981}
982
983/// A diagnostic function for dumping crate metadata to an output stream.
984pub fn list_file_metadata(
985 target: &Target,
986 path: &Path,
987 metadata_loader: &dyn MetadataLoader,
988 out: &mut dyn Write,
989 ls_kinds: &[String],
990 cfg_version: &'static str,
991) -> IoResult<()> {
992 let flavor = get_flavor_from_path(path);
993 match get_metadata_section(target, flavor, path, metadata_loader, cfg_version, None) {
994 Ok(metadata) => metadata.list_crate_metadata(out, ls_kinds),
995 Err(msg) => write!(out, "{msg}\n"),
996 }
997}
998
999fn get_flavor_from_path(path: &Path) -> CrateFlavor {
1000 let filename = path.file_name().unwrap().to_str().unwrap();
1001
1002 if filename.ends_with(".rlib") {
1003 CrateFlavor::Rlib
1004 } else if filename.ends_with(".rmeta") {
1005 CrateFlavor::Rmeta
1006 } else {
1007 CrateFlavor::Dylib
1008 }
1009}
1010
1011// ------------------------------------------ Error reporting -------------------------------------
1012
1013#[derive(Clone, Debug)]
1014struct CrateMismatch {
1015 path: PathBuf,
1016 got: String,
1017}
1018
1019#[derive(Clone, Debug, Default)]
1020pub(crate) struct CrateRejections {
1021 via_hash: Vec<CrateMismatch>,
1022 via_triple: Vec<CrateMismatch>,
1023 via_kind: Vec<CrateMismatch>,
1024 via_version: Vec<CrateMismatch>,
1025 via_filename: Vec<CrateMismatch>,
1026 via_invalid: Vec<CrateMismatch>,
1027}
1028
1029/// Candidate rejection reasons collected during crate search.
1030/// If no candidate is accepted, then these reasons are presented to the user,
1031/// otherwise they are ignored.
1032#[derive(Debug)]
1033pub(crate) struct CombinedLocatorError {
1034 crate_name: Symbol,
1035 dep_root: Option<CratePaths>,
1036 triple: TargetTuple,
1037 dll_prefix: String,
1038 dll_suffix: String,
1039 crate_rejections: CrateRejections,
1040}
1041
1042#[derive(Debug)]
1043pub(crate) enum CrateError {
1044 NonAsciiName(Symbol),
1045 ExternLocationNotExist(Symbol, PathBuf),
1046 ExternLocationNotFile(Symbol, PathBuf),
1047 MultipleCandidates(Symbol, CrateFlavor, Vec<PathBuf>),
1048 FullMetadataNotFound(Symbol, CrateFlavor),
1049 SymbolConflictsCurrent(Symbol),
1050 StableCrateIdCollision(Symbol, Symbol),
1051 DlOpen(String, String),
1052 DlSym(String, String),
1053 LocatorCombined(Box<CombinedLocatorError>),
1054 NotFound(Symbol),
1055}
1056
1057enum MetadataError<'a> {
1058 /// The file was missing.
1059 NotPresent(&'a Path),
1060 /// The file was present and invalid.
1061 LoadFailure(String),
1062 /// The file was present, but compiled with a different rustc version.
1063 VersionMismatch { expected_version: String, found_version: String },
1064}
1065
1066impl fmt::Display for MetadataError<'_> {
1067 fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
1068 match self {
1069 MetadataError::NotPresent(filename) => {
1070 f.write_str(&format!("no such file: '{}'", filename.display()))
1071 }
1072 MetadataError::LoadFailure(msg) => f.write_str(msg),
1073 MetadataError::VersionMismatch { expected_version, found_version } => {
1074 f.write_str(&format!(
1075 "rustc version mismatch. expected {}, found {}",
1076 expected_version, found_version,
1077 ))
1078 }
1079 }
1080 }
1081}
1082
1083impl CrateError {
1084 pub(crate) fn report(self, sess: &Session, span: Span, missing_core: bool) {
1085 let dcx = sess.dcx();
1086 match self {
1087 CrateError::NonAsciiName(crate_name) => {
1088 dcx.emit_err(errors::NonAsciiName { span, crate_name });
1089 }
1090 CrateError::ExternLocationNotExist(crate_name, loc) => {
1091 dcx.emit_err(errors::ExternLocationNotExist { span, crate_name, location: &loc });
1092 }
1093 CrateError::ExternLocationNotFile(crate_name, loc) => {
1094 dcx.emit_err(errors::ExternLocationNotFile { span, crate_name, location: &loc });
1095 }
1096 CrateError::MultipleCandidates(crate_name, flavor, candidates) => {
1097 dcx.emit_err(errors::MultipleCandidates { span, crate_name, flavor, candidates });
1098 }
1099 CrateError::FullMetadataNotFound(crate_name, flavor) => {
1100 dcx.emit_err(errors::FullMetadataNotFound { span, crate_name, flavor });
1101 }
1102 CrateError::SymbolConflictsCurrent(root_name) => {
1103 dcx.emit_err(errors::SymbolConflictsCurrent { span, crate_name: root_name });
1104 }
1105 CrateError::StableCrateIdCollision(crate_name0, crate_name1) => {
1106 dcx.emit_err(errors::StableCrateIdCollision { span, crate_name0, crate_name1 });
1107 }
1108 CrateError::DlOpen(path, err) | CrateError::DlSym(path, err) => {
1109 dcx.emit_err(errors::DlError { span, path, err });
1110 }
1111 CrateError::LocatorCombined(locator) => {
1112 let crate_name = locator.crate_name;
1113 let add_info = match &locator.dep_root {
1114 None => String::new(),
1115 Some(r) => format!(" which `{}` depends on", r.name),
1116 };
1117 if !locator.crate_rejections.via_filename.is_empty() {
1118 let mismatches = locator.crate_rejections.via_filename.iter();
1119 for CrateMismatch { path, .. } in mismatches {
1120 dcx.emit_err(errors::CrateLocationUnknownType { span, path, crate_name });
1121 dcx.emit_err(errors::LibFilenameForm {
1122 span,
1123 dll_prefix: &locator.dll_prefix,
1124 dll_suffix: &locator.dll_suffix,
1125 });
1126 }
1127 }
1128 let mut found_crates = String::new();
1129 if !locator.crate_rejections.via_hash.is_empty() {
1130 let mismatches = locator.crate_rejections.via_hash.iter();
1131 for CrateMismatch { path, .. } in mismatches {
1132 found_crates.push_str(&format!(
1133 "\ncrate `{}`: {}",
1134 crate_name,
1135 path.display()
1136 ));
1137 }
1138 if let Some(r) = locator.dep_root {
1139 for path in r.source.paths() {
1140 found_crates.push_str(&format!(
1141 "\ncrate `{}`: {}",
1142 r.name,
1143 path.display()
1144 ));
1145 }
1146 }
1147 dcx.emit_err(errors::NewerCrateVersion {
1148 span,
1149 crate_name,
1150 add_info,
1151 found_crates,
1152 });
1153 } else if !locator.crate_rejections.via_triple.is_empty() {
1154 let mismatches = locator.crate_rejections.via_triple.iter();
1155 for CrateMismatch { path, got } in mismatches {
1156 found_crates.push_str(&format!(
1157 "\ncrate `{}`, target triple {}: {}",
1158 crate_name,
1159 got,
1160 path.display(),
1161 ));
1162 }
1163 dcx.emit_err(errors::NoCrateWithTriple {
1164 span,
1165 crate_name,
1166 locator_triple: locator.triple.tuple(),
1167 add_info,
1168 found_crates,
1169 });
1170 } else if !locator.crate_rejections.via_kind.is_empty() {
1171 let mismatches = locator.crate_rejections.via_kind.iter();
1172 for CrateMismatch { path, .. } in mismatches {
1173 found_crates.push_str(&format!(
1174 "\ncrate `{}`: {}",
1175 crate_name,
1176 path.display()
1177 ));
1178 }
1179 dcx.emit_err(errors::FoundStaticlib {
1180 span,
1181 crate_name,
1182 add_info,
1183 found_crates,
1184 });
1185 } else if !locator.crate_rejections.via_version.is_empty() {
1186 let mismatches = locator.crate_rejections.via_version.iter();
1187 for CrateMismatch { path, got } in mismatches {
1188 found_crates.push_str(&format!(
1189 "\ncrate `{}` compiled by {}: {}",
1190 crate_name,
1191 got,
1192 path.display(),
1193 ));
1194 }
1195 dcx.emit_err(errors::IncompatibleRustc {
1196 span,
1197 crate_name,
1198 add_info,
1199 found_crates,
1200 rustc_version: rustc_version(sess.cfg_version),
1201 });
1202 } else if !locator.crate_rejections.via_invalid.is_empty() {
1203 let mut crate_rejections = Vec::new();
1204 for CrateMismatch { path: _, got } in locator.crate_rejections.via_invalid {
1205 crate_rejections.push(got);
1206 }
1207 dcx.emit_err(errors::InvalidMetadataFiles {
1208 span,
1209 crate_name,
1210 add_info,
1211 crate_rejections,
1212 });
1213 } else {
1214 let error = errors::CannotFindCrate {
1215 span,
1216 crate_name,
1217 add_info,
1218 missing_core,
1219 current_crate: sess
1220 .opts
1221 .crate_name
1222 .clone()
1223 .unwrap_or_else(|| "<unknown>".to_string()),
1224 is_nightly_build: sess.is_nightly_build(),
1225 profiler_runtime: Symbol::intern(&sess.opts.unstable_opts.profiler_runtime),
1226 locator_triple: locator.triple,
1227 is_ui_testing: sess.opts.unstable_opts.ui_testing,
1228 };
1229 // The diagnostic for missing core is very good, but it is followed by a lot of
1230 // other diagnostics that do not add information.
1231 if missing_core {
1232 dcx.emit_fatal(error);
1233 } else {
1234 dcx.emit_err(error);
1235 }
1236 }
1237 }
1238 CrateError::NotFound(crate_name) => {
1239 let error = errors::CannotFindCrate {
1240 span,
1241 crate_name,
1242 add_info: String::new(),
1243 missing_core,
1244 current_crate: sess
1245 .opts
1246 .crate_name
1247 .clone()
1248 .unwrap_or_else(|| "<unknown>".to_string()),
1249 is_nightly_build: sess.is_nightly_build(),
1250 profiler_runtime: Symbol::intern(&sess.opts.unstable_opts.profiler_runtime),
1251 locator_triple: sess.opts.target_triple.clone(),
1252 is_ui_testing: sess.opts.unstable_opts.ui_testing,
1253 };
1254 // The diagnostic for missing core is very good, but it is followed by a lot of
1255 // other diagnostics that do not add information.
1256 if missing_core {
1257 dcx.emit_fatal(error);
1258 } else {
1259 dcx.emit_err(error);
1260 }
1261 }
1262 }
1263 }
1264}