byte/char string representation (Was: [PATCH] Fix byte char and byte string lexing code)

[Mark Wielaard <mark@klomp.org>]

On Thu, Sep 23, 2021 at 04:10:59PM +0200, Arthur Cohen wrote:
> > Something I was thinking about outside of the scope of that patch was
> about the utf8 how do they get represented? Is it some kind of wchar_t?
> 
> Do you mean in C++ or in rustc? In rustc, they are represented as Unicode
> Scalar Values which are 4 bytes wide.
> 
> From the docs over here: [https://doc.rust-lang.org/std/primitive.char.html]
> 
> So I'm assuming they could be represented as `int32_t`s which would also
> make sense for the check

Yes, for rust characters a 32bit type (I would pick uint32_t or maybe
char32_t) makes sense, since chars in rust are (almost) equal to
unicode code points (technically only 21 bits are used). But not
really, it is a Unicode scalar value, which excludes high-surrogate
and low-surrogate code points and so the only valid values are 0x0 to
0xD7FF and 0xE000 to 0x10FFFF.

We should not use the C type wchar_t, because wchar_t is
implementation defined and can be 16 or 32 bits.

See also https://doc.rust-lang.org/reference/types/textual.html

But utf8 strings are made up of u8 "utf8 chars". You need 1 to 4 utf8
chars to encode a code point. https://en.wikipedia.org/wiki/UTF-8
We can use c++ strings made up of (8 bit) chars for that.

Our lexer should make sure we only accept valid rust characters or
utf-8 sequences.

Note that the above doesn't hold for "byte chars" (b'A') or "byte
strings" (b"abc"). Those are really just u8 or [u8] arrays which hold
bytes (0x0 to 0xff).

We currently get the type for byte strings wrong. We pretend they are
&str, but they really should be &[u8].

I tried to fix that with the following:

diff --git a/gcc/rust/typecheck/rust-hir-type-check-expr.h b/gcc/rust/typecheck/rust-hir-type-check-expr.h
index fe8973a4d81..b0dd1c3ff2c 100644
--- a/gcc/rust/typecheck/rust-hir-type-check-expr.h
+++ b/gcc/rust/typecheck/rust-hir-type-check-expr.h
@@ -609,15 +609,42 @@ public:
 	break;
 
 	case HIR::Literal::LitType::BYTE_STRING: {
-	  /* We just treat this as a string, but it really is an arraytype of
-	     u8. It isn't in UTF-8, but really just a byte array.  */
-	  TyTy::BaseType *base = nullptr;
-	  auto ok = context->lookup_builtin ("str", &base);
+	  /* This is an arraytype of u8 reference (&[u8;size]). It isn't in
+	     UTF-8, but really just a byte array. Code to construct the array
+	     reference copied from ArrayElemsValues and ArrayType. */
+	  TyTy::BaseType *u8;
+	  auto ok = context->lookup_builtin ("u8", &u8);
 	  rust_assert (ok);
 
+	  auto crate_num = mappings->get_current_crate ();
+	  Analysis::NodeMapping mapping (crate_num, UNKNOWN_NODEID,
+					 mappings->get_next_hir_id (crate_num),
+					 UNKNOWN_LOCAL_DEFID);
+
+	  /* Capacity is the size of the string (number of chars).
+	     It is a constant, but for fold it to get a BExpression.  */
+	  std::string capacity_str = std::to_string (expr.as_string ().size ());
+	  HIR::LiteralExpr literal_capacity (mapping, capacity_str,
+					     HIR::Literal::LitType::INT,
+					     PrimitiveCoreType::CORETYPE_USIZE,
+					     expr.get_locus ());
+
+	  // mark the type for this implicit node
+	  context->insert_type (mapping,
+				new TyTy::USizeType (mapping.get_hirid ()));
+
+	  Bexpression *capacity
+	    = ConstFold::ConstFoldExpr::fold (&literal_capacity);
+
+	  TyTy::ArrayType *array
+	    = new TyTy::ArrayType (expr.get_mappings ().get_hirid (), capacity,
+				   TyTy::TyVar (u8->get_ref ()));
+
+	  context->insert_type (expr.get_mappings (), array);
+
 	  infered
 	    = new TyTy::ReferenceType (expr.get_mappings ().get_hirid (),
-				       TyTy::TyVar (base->get_ref ()), false);
+				       TyTy::TyVar (array->get_ref ()), false);
 	}
 	break;
 
But that looks more complicated than is probably necessary and it
doesn't work. When the type checker wants to print this type
ReferenceType.as_string () goes into a loop for some reason.

Can anybody see what is wrong with the above?

Cheers,

Mark