What a PyTorch tensor really is

pgmesa · · 30 min read

Part of Rebuilding PyTorch (2 of 8)
  1. Why I rebuilt PyTorch's core from scratch
  2. What a PyTorch tensor really is
  3. Interlude: What the CPU does when your code reads memory
  4. How PyTorch walks a tensor of any shape
  5. How PyTorch prepares the loop of every operation
  6. How a PyTorch kernel uses the vector registers of every CPU
  7. How PyTorch sums a tensor, and splits the work between threads
  8. How PyTorch computes gradients
  9. How PyTorch runs the same operations on a GPU

Before we can ask why PyTorch is fast, we need to know what it is working on. So the question of this first article looks almost too simple: what is a tensor? We use them every day as a block of numbers with a shape, but that is not what PyTorch keeps in memory, and the difference explains why some operations are free and others copy, what “contiguous” means, and what the loops of the next articles have to deal with.

Before you start. This article only assumes that you have used PyTorch tensors and indexed them. The C++ snippets are short, and the text says what each one does.

This is the path the article follows:

  1. Device, layout, dtype.
  2. A tensor is not its numbers.
  3. The type and the device are values.
  4. From an index to a position.
  5. Views.
  6. A stride of 0.
  7. Sharing has consequences.
  8. Who frees the memory.
  9. What contiguous really means.
  10. view and reshape.
  11. How much does it cost?
  12. What Raxis does differently.

And at the end, where this leaves us.

1. Device, layout, dtype

Besides its shape, every PyTorch tensor answers three questions:

>>> x = torch.zeros(3, 4)
>>> x.dtype, x.device, x.layout
(torch.float32, device(type='cpu'), torch.strided)

Edward Yang calls these three the trinity of the tensor in “PyTorch internals”: each combination of them is a different kind of tensor, and each may need its own code.

The first two are familiar. The layout is the one we hardly ever see, because almost every tensor has the same one, torch.strided: the numbers in one block of memory, and for each dimension a stride that says how to find them. It is the subject of this article. PyTorch has others, for data that a block of memory would waste:

Each keeps its own indices or offsets next to the values, and needs kernels of its own, so many operations support only some of them.

Raxis has only the strided layout, and everything in this series is about it: views, contiguity, and the loops and kernels of the next articles. It is the layout that nearly all real code uses, and the one where a view costs nothing.

To see what that means, look at the second line of this:

x = torch.zeros(10_000, 10_000)   # 400 MB
y = x[1:3]                        # 20,000 numbers

It takes about a microsecond. If x were 32 x 32, it would also take about a microsecond. So whatever y is, it did not copy those 20,000 numbers anywhere.

Then what is y? And what is x, for that matter? Everything else in this series, the loops, the kernels and even autograd, stands on the answer, so this is where we start.

2. A tensor is not its numbers

There are three layers. This is what the two tensors above look like in memory:

Two tensors over one storagex and y are two TensorImpls with their own shape, strides and offset. Both point to the same StorageImpl of 100 million numbers; y reads rows 1 and 2, starting at offset 10000.x TensorImplshape (10000, 10000)strides (10000, 1)offset 0storagey = x[1:3] TensorImplshape (2, 10000)strides (10000, 1)offset 10000storageStorageImplpointer, size, how to free itrow 0row 1row 2rows 3 to 999901000030000100,000,000400 MB: one row of 10000 float32 after another
Two TensorImpls, one StorageImpl. Each points at the same memory, and its offset says where its first number is: y is a window of two rows inside the 400 MB of x.

Why split the last two? Because of y. Many TensorImpls can read one buffer in different ways, and the buffer is freed only when the last of them goes away, because the StorageImpl is reference-counted too. x[1:3] creates one new TensorImpl with a different shape and offset, and no new storage.

This is the part of Raxis’s TensorImpl that matters for this article:

std::shared_ptr<StorageImpl> storage_;
int64_t storage_offset_ = 0;

DimVector shape_;
DimVector strides_;
Dtype dtype_;

PyTorch has the same fields, and more around them: a set of dispatch keys, a slot for its Python object, autograd metadata. Those serve other parts of the system, and we will get to some of them. But the description of the numbers is exactly this.

3. The type and the device are values

Two of the properties of the first section deserve a closer look, because they are values, decided when the program runs, and in C++ that is not the obvious choice.

The dtype in Raxis is a one-byte enum wrapped in a small class that knows the size of an element and the name of the type (raxisdtype.h, raxisDtype). PyTorch has the same two pieces, with more types: the enum is ScalarType (pytorchScalarType.h), and what the TensorImpl stores (pytorchTensorImpl.h) is a TypeMeta, a 16-bit index into a table that holds the size and the name of each type (pytorchtypeid.h, pytorchtypeid.h). The device is a type and an index (pytorchDevice.h, raxisdevice.h): DeviceType::CUDA with index 0 is cuda:0. A Raxis TensorImpl does not even store its device: it asks its storage, whose DataPtr carries it, as we will see (raxistensor_impl.cpp). PyTorch keeps a copy in the TensorImpl as well (pytorchTensorImpl.h).

If you come from C++, you might wonder why the type is not a template parameter, Tensor<float> and Tensor<int64_t>. The compiler would know the type of every element, and every loop would be typed for free. My first tensor in C++, in 2024, was written that way, and it does not hold up in a library like this one:

So the type is a value everywhere, and becomes a template parameter only where that pays off: inside the kernel, once per operation. This is how Raxis’s add picks its type (raxisbinary_ops_kernel.cpp):

iter.common_dtype().dispatch_all_types([&]<typename scalar_t>(std::type_identity<scalar_t>) {
    cpu_kernel_vec(iter,
        [](scalar_t a, scalar_t b) -> scalar_t { return a + b; },
        [](Vectorized<scalar_t> a, Vectorized<scalar_t> b) { return a + b; });
});

dispatch_all_types is a switch over the dtype that calls the lambda with each C++ type (raxisdtype.h), so the loop inside is compiled once per type, and each version is fully typed. PyTorch does the same with its AT_DISPATCH macros (pytorchDispatch.h). The switch runs once per call to a + b, not once per element. How type promotion picks the type of the result is in the TensorIterator article, and the device is chosen in a similar way one level up, as the CUDA article shows.

4. From an index to a position

For the rest of the article, let’s use a tensor small enough to see:

x = torch.arange(12).reshape(3, 4)
tensor([[ 0,  1,  2,  3],
        [ 4,  5,  6,  7],
        [ 8,  9, 10, 11]])

Its dtype is int64, the default for integers, so each number takes 8 bytes and the storage is 96 bytes long. Those rows are only how it is printed. The memory is a flat line of 12 numbers, and I chose values that match their positions in it, so that each number tells you where it lives:

0 1 2 3 4 5 6 7 8 9 10 11

A stride says how far to jump in that line when the index of a dimension grows by one. To go to the next row of x you jump 4 positions, and to the next column you jump 1, so its strides are (4, 1). With that, any element is one sum away:

position = offset + i * strides[0] + j * strides[1]

x[1, 2]  = 0      + 1 * 4          + 2 * 1           = 6

With more dimensions there are more terms, and nothing else changes. The number of dimensions does not matter to the formula, which will be very convenient in the next article.

The strides of a new tensor come from its shape, going from the last dimension to the first: the last one is 1, and each of the others is the previous stride times the previous size (raxisshape_utils.cpp). This is called row-major order, because the elements of a row end up next to each other.

And the offset is where the first element is. Strides and offset count elements, not bytes, so this is where the dtype comes in: the pointer to the first element, what PyTorch calls data_ptr, is the start of the storage plus the offset times the size of one element, 8 bytes for x (pytorchTensorImpl.h, raxistensor_impl.cpp). That is why a tensor is really a pointer, a shape and strides.

One detail that will catch you if you read both libraries: PyTorch counts its strides and its offset in elements, and NumPy counts its strides in bytes. The same x has strides (4, 1) in PyTorch and (32, 8) in NumPy: 4 elements of 8 bytes to the next row, and one element of 8 bytes to the next column.

The snippets here are PyTorch’s, and they give the same numbers in Raxis.

5. Views

Now look at what an indexing operation has to do. Nothing in the formula mentions the data, so to read a part of x differently, it is enough to write new numbers in a new TensorImpl. Try it:

shape
strides
offset
contiguous

The tensor you see

The storage, always the same 12 numbers

Choose a view, then point at one of its elements. Highlighted cells are the ones the view reads. A number next to a cell says how many elements of the view read it.
ViewShapeStridesOffset
x(3, 4)(4, 1)0
x[1](4,)(1,)4
x[1:3](2, 4)(4, 1)4
x[:, 1](3,)(4,)1
x[:, ::2](3, 2)(4, 2)0
x.transpose(0, 1)(4, 3)(1, 4)0

Each of these operations is a few lines of arithmetic:

This is the slice in Raxis (raxistensor_shape.cpp), without the checks:

int64_t new_offset = self.storage_offset() + start * old_strides[dim];

DimVector new_shape(old_shape);
DimVector new_strides(old_strides);
new_shape[dim] = new_dim_size;
new_strides[dim] = old_strides[dim] * step;

return make_view(self, new_offset, new_shape, new_strides);

PyTorch does the same arithmetic in pytorchTensorShape.cpp, with more branches around it, and creates the result by going through its dispatcher (the layer that routes every call to the right implementation, which Raxis replaces with direct calls) to as_strided. In Raxis, raxismake_view creates the new TensorImpl directly.

6. A stride of 0

There is one more view, and it deserves its own section. Take a column of x, with shape (3, 1), and expand it:

>>> c = x[:, :1]
>>> e = c.expand(3, 4)
>>> e
tensor([[0, 0, 0, 0],
        [4, 4, 4, 4],
        [8, 8, 8, 8]])
>>> e.stride()
(4, 0)

That tensor has 12 elements and no memory of its own. Its second stride is 0, so moving along a row moves nowhere: all the columns read the same position. Select x[:, :1].expand(3, 4) in the figure above and you will see it, three cells read four times each.

expand is about nothing more than this: every dimension that grows from size 1 gets a stride of 0 (pytorchExpandUtils.cpp, raxistensor_shape.cpp).

So a repeated dimension is not a special case, it is one more stride, and the formula above does not need to change. Keep this in mind for later: broadcasting, when two tensors of different shapes meet in a + b, comes down to this.

7. Sharing has consequences

If two tensors read the same memory, writing through one changes the other:

>>> x = torch.arange(12).reshape(3, 4)
>>> row = x[1]
>>> row[2] = 99
>>> x
tensor([[ 0,  1,  2,  3],
        [ 4,  5, 99,  7],
        [ 8,  9, 10, 11]])

You probably knew that one. These may be less expected.

Writing to an expanded tensor writes to all of its copies, because they are not copies:

>>> x = torch.arange(12).reshape(3, 4)
>>> e = x[:, :1].expand(3, 4)
>>> e[0, 2] = 99
>>> e
tensor([[99, 99, 99, 99],
        [ 4,  4,  4,  4],
        [ 8,  8,  8,  8]])

I asked to change one element and four changed, along with x[0, 0]. An in-place operation over the whole tensor does not let you get that far, because each of those positions would be written several times:

>>> e.add_(1)
RuntimeError: unsupported operation: more than one element of the written-to tensor
refers to a single memory location. Please clone() the tensor before performing the operation.

NumPy takes another path here: np.broadcast_to returns a read-only array, so you cannot make the first mistake either.

A small view keeps the whole storage alive. The storage is freed when nothing refers to it, and a view refers to all of it:

big = torch.zeros(10_000_000)    # 40 MB
head = big[:2]
del big                          # the 40 MB are still there

Worse, torch.save writes storages, not views. Saving that head of 2 numbers produces a file of 40 MB, and saving head.clone() a file of 1.6 KB. If you keep a small piece of something big, clone it.

Overwriting a view can break autograd. This one will matter when we get to autograd. To compute gradients, autograd saves some tensors during the forward pass. exp, for example, saves its result, because the derivative of exp(x) is exp(x) itself. If something overwrites those values before the backward pass, the gradient would be wrong, and with views that something does not have to be the saved tensor:

>>> a = torch.ones(3, requires_grad=True)
>>> b = a.exp()          # exp saves b for the backward pass
>>> b[:1].mul_(2)        # a view of b writes into the memory of b
>>> b.sum().backward()
RuntimeError: one of the variables needed for gradient computation has been modified
by an inplace operation: [torch.FloatTensor [3]], which is output 0 of Exp, is at
version 1; expected version 0 instead. ...

PyTorch catches it with a version counter. Every in-place operation increments it, autograd writes down the version of each tensor it saves, and refuses to run the backward pass if the version has changed since. For this to work, a view and its base must share the same counter, or writing through b[:1] would not change the version of b. PyTorch keeps the counter in the TensorImpl and gives every view the same one as its base (pytorchTensorImpl.h). Raxis keeps it in the StorageImpl (raxisstorage_impl.h), which every view already shares. The rest of the story, how autograd saves and checks, belongs to the autograd articles.

8. Who frees the memory

So a storage lives as long as something refers to it. When the last reference finally goes, who frees its memory, and how? It is less obvious than it looks, because the memory of a tensor comes from very different places, and each one has to be given back in its own way.

The memory comes from an allocator, the piece of the library that hands out memory and takes it back:

So the storage keeps, next to the pointer, what has to be released and how. That is PyTorch’s DataPtr (pytorchAllocator.h, raxisdata_ptr.h), built on a UniqueVoidPtr (pytorchUniqueVoidPtr.h, raxisunique_void_ptr.h):

class UniqueVoidPtr {
    void* data_;                                // where the numbers start
    std::unique_ptr<void, DeleterFnPtr> ctx_;   // what to release, and the function that does it
};

class DataPtr {
    UniqueVoidPtr ptr_;
    Device device_;                             // where that memory lives: CPU, cuda:0, ...
};

The data and the context are separate because they are not always the same thing:

This is the whole chain for torch.from_numpy(a), from the tensor down to the two pointers:

From a Tensor to the memory of a NumPy arrayA Tensor points to its TensorImpl, which points to a StorageImpl. The StorageImpl holds a DataPtr with the device and a UniqueVoidPtr. Its data pointer goes to the numbers of the NumPy array, and its context to a Python object that holds a reference to the array, which is what the deleter releases. With the CPU allocator, data and context are the same pointer.Tensor t = torch.from_numpy(a)impl_TensorImplshape, strides, offset, dtypestorage_StorageImplnbytes_DataPtrdevice_ cpuUniqueVoidPtrdata_ctx_ + deletera ndarraya Python object, owned by NumPyits bufferthe numbers of apy::objecta reference to a: keeps it aliveThe deleter deletes the py::object: it drops thereference, and NumPy frees the numbers later.CPU allocatordata_ctx_the numbers
What torch.from_numpy(a) builds. The data pointer goes to the numbers, and the context to a reference to the array, so freeing the tensor only drops that reference. With the CPU allocator, both point to the same block, and the deleter frees it.

This is what lets one kind of tensor hold memory from anywhere. The StorageImpl never asks where its memory came from: when the last reference to it goes away, the unique_ptr calls the deleter it was given, and the right owner gets its memory back. And because the device travels with the pointer, nothing reads a GPU address as if it were CPU memory. How each device picks its allocator is in the CUDA article.

9. What contiguous really means

is_contiguous() looks easy, and it is easy to read too much into it. Its test is only this (pytorchContiguity.h, raxisshape_utils.cpp):

int64_t expected_stride = 1;
for (int i = ndim - 1; i >= 0; --i) {
    if (shape[i] == 1) continue;   // never advances, so its stride does not matter
    if (strides[i] != expected_stride) return false;
    expected_stride *= shape[i];
}
return true;

In words: walking the tensor in its logical order means walking memory one position at a time, with no jumps. Look at what it does not say:

TensorContiguousWhy
x[1:3]YesIt starts at offset 4 and leaves part of the storage out, and that does not matter
x[1:2], shape (1, 4)YesThe size 1 dimension does not count
x[:, 1:3]NoAfter each row, it has to jump over the other columns
x.transpose(0, 1)NoIt covers its block of memory with nothing left out, but in another order
x[:, :1], shape (3, 1)NoOnly one column, and each row is 4 positions away

So contiguous does not mean “the whole storage”, nor “starts at the beginning”, nor even “its numbers fill a block of memory with nothing in between”: the transpose does that and is still not contiguous. It is a statement about order. Any tensor can be walked with its pointer and its strides, but a contiguous one is also a plain array: its numbers are p[0] to p[numel - 1], one loop with one index, which the compiler can vectorize and a copy can do with a single memcpy. Any other tensor needs the shape and strides of every dimension to find the next number, and how to do that fast is the subject of the next article.

And it is one order in particular: row-major, also called C order, the default in PyTorch and NumPy. Fortran stores matrices the other way, column by column, and so do BLAS and LAPACK, which come from it. That is column-major, or Fortran order. NumPy keeps track of both (a.flags['C_CONTIGUOUS'] and a.flags['F_CONTIGUOUS']), while PyTorch’s is_contiguous() checks only the first. The transpose of a row-major matrix is a column-major one. That is why x.transpose(0, 1) is not contiguous, and also why a matrix multiplication can give it to BLAS as it is, telling BLAS that the matrix is transposed instead of copying it (pytorchLinearAlgebra.cpp, raxislinear_algebra.cpp).

When something does need that order, x.contiguous() returns the same tensor if it already has it, without copying, and a copy in the right order otherwise (raxistensor.cpp). That second case is the first time in this article that a number actually moves.

10. view and reshape

These two look like synonyms, and the difference comes from all of the above. view never copies, so it fails if the result cannot be written with strides. reshape tries the same thing and, if it cannot, copies:

>>> t = x.transpose(0, 1)    # shape (4, 3), strides (1, 4)
>>> t.view(12)
RuntimeError: view size is not compatible with input tensor's size and stride
(at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
>>> t.reshape(12)
tensor([ 0,  4,  8,  1,  5,  9,  2,  6, 10,  3,  7, 11])

So reshape gives you a tensor that sometimes shares memory with the original and sometimes does not, depending on the strides. If you are going to write to the result, that is worth remembering.

It may surprise you that view does not need a contiguous tensor. It only needs each new dimension to fit in the strides:

>>> x[:, ::2].view(6)                  # not contiguous, and it works
tensor([ 0,  2,  4,  6,  8, 10])
>>> t.view(2, 2, 3).stride()           # neither is this
(2, 1, 4)

x[:, ::2] has shape (3, 2) and strides (4, 2). Its elements are evenly spaced, two positions apart, because the outer stride of 4 is exactly the inner stride of 2 times the inner size of 2, so the two dimensions can merge into one with stride 2. In t, that check fails: merging needs an outer stride of 4 * 3 = 12, and it is 1. Splitting a dimension, as in t.view(2, 2, 3), always works. This is what computeStride works out in pytorchTensorUtils.cpp, and raxiscompute_view_strides in Raxis.

A last limit, in case you come from NumPy: x[::-1] is a negative stride, and PyTorch does not have them. NumPy gives you strides (-32, 8) with no copy, and PyTorch raises an error (its slice has a TODO: support negative strides), so reversing a tensor goes through flip, which copies.

11. How much does it cost?

This is all measured with raxisbench_views.py, on a Ryzen 7 9700X with 8 threads, in November 2026, with PyTorch 2.14.1 and Raxis 0.5.0, on Windows 11 25H2, on float32 square tensors of three sizes:

Operation32 x 321000 x 100010000 x 10000
x[1:3], Raxis1.2 us1.1 us1.1 us
x[1:3], PyTorch1.1 us1.1 us1.5 us
x.transpose(0, 1), Raxis0.6 us0.9 us0.7 us
x.transpose(0, 1), PyTorch0.9 us0.9 us0.9 us
x[:, :1].expand(n, n), Raxis2.5 us2.6 us2.5 us
x[:, :1].expand(n, n), PyTorch2.4 us2.4 us2.4 us
x.clone(), Raxis0.9 us16 us22.5 ms
x.clone(), PyTorch1.1 us16 us24.6 ms
x.transpose(0, 1).contiguous(), Raxis1.1 us105 us37 ms
x.transpose(0, 1).contiguous(), PyTorch1.7 us507 us97 ms

The views take the same time at any size, as the formula promised. They are not free: a microsecond is a new TensorImpl and a Python object around it, so a Python loop that takes a view per row pays that for every row. That is one reason why kernels do not work that way. But between 32 x 32 and 10000 x 10000 there are five orders of magnitude in the data, and none in the time.

The last two rows are the same copy done in a bad order, and that costs more than a plain clone(). The gap between the two libraries there has a concrete reason: for two dimensional transposes, PyTorch has a special path in pytorchCopy.cpp that copies by blocks on a single thread, and Raxis goes through its general copy, in parallel. On one thread, at 1000 x 1000, they are much closer: 0.39 ms against 0.51 ms.

12. What Raxis does differently

The model is the same, so the differences are small. Each one has its reasons, and none of them changes anything you have read here:

Where this leaves us

We can now describe any tensor with a few numbers, and we know how those numbers change without touching the data. But we have paid for that. A stride can be anything, 1, 4 or 0. The number of dimensions is known only at runtime. So how do you write a loop that adds two of these tensors, and how do you make it fast?

That is the question of How PyTorch walks a tensor of any shape. It is the question I most wanted to answer when I started, and I found little written about it in depth. Before it comes an interlude, What the CPU does when your code reads memory, with the hardware background that the rest of the series refers to: caches, SIMD and threads. If you know it already, you can skip it, or come back to it whenever an article links there.

Further reading

If something is not clear, or you think I got something wrong, leave a comment.