What a PyTorch tensor really is
pgmesa · · 30 min read
Part of Rebuilding PyTorch (2 of 8)
- Why I rebuilt PyTorch's core from scratch
- What a PyTorch tensor really is
- Interlude: What the CPU does when your code reads memory
- How PyTorch walks a tensor of any shape
- How PyTorch prepares the loop of every operation
- How a PyTorch kernel uses the vector registers of every CPU
- How PyTorch sums a tensor, and splits the work between threads
- How PyTorch computes gradients
- How PyTorch runs the same operations on a GPU
Before we can ask why PyTorch is fast, we need to know what it is working on. So the question of this first article looks almost too simple: what is a tensor? We use them every day as a block of numbers with a shape, but that is not what PyTorch keeps in memory, and the difference explains why some operations are free and others copy, what “contiguous” means, and what the loops of the next articles have to deal with.
Before you start. This article only assumes that you have used PyTorch tensors and indexed them. The C++ snippets are short, and the text says what each one does.
This is the path the article follows:
And at the end, where this leaves us.
1. Device, layout, dtype
Besides its shape, every PyTorch tensor answers three questions:
>>> x = torch.zeros(3, 4)
>>> x.dtype, x.device, x.layout
(torch.float32, device(type='cpu'), torch.strided)
- The dtype is what kind of number each element is:
float32,int64,bool, … - The device is where the numbers live: the CPU, or one particular GPU, such as
cuda:0. - The layout is how they are arranged in memory.
Edward Yang calls these three the trinity of the tensor in “PyTorch internals”: each combination of them is a different kind of tensor, and each may need its own code.
The first two are familiar. The layout is the one we hardly ever see, because almost every tensor has the same one, torch.strided: the numbers in one block of memory, and for each dimension a stride that says how to find them. It is the subject of this article. PyTorch has others, for data that a block of memory would waste:
torch.sparse_cookeeps only the values that are not zero, together with their coordinates.torch.sparse_csr,sparse_csc,sparse_bsrandsparse_bsccompress the rows, the columns, or blocks of them, as the libraries for sparse matrices do.torch._mkldnnis the blocked format that oneDNN uses inside its CPU convolutions.torch.jaggedholds nested tensors, whose rows can have different lengths.
Each keeps its own indices or offsets next to the values, and needs kernels of its own, so many operations support only some of them.
Raxis has only the strided layout, and everything in this series is about it: views, contiguity, and the loops and kernels of the next articles. It is the layout that nearly all real code uses, and the one where a view costs nothing.
To see what that means, look at the second line of this:
x = torch.zeros(10_000, 10_000) # 400 MB
y = x[1:3] # 20,000 numbers
It takes about a microsecond. If x were 32 x 32, it would also take about a microsecond. So whatever y is, it did not copy those 20,000 numbers anywhere.
Then what is y? And what is x, for that matter? Everything else in this series, the loops, the kernels and even autograd, stands on the answer, so this is where we start.
2. A tensor is not its numbers
There are three layers. This is what the two tensors above look like in memory:
TensorImpls, one StorageImpl. Each points at the same memory, and its offset says where its first number is: y is a window of two rows inside the 400 MB of x.- The
Tensoryou hold is a handle: one reference-counted pointer to aTensorImpl. Copying it copies the pointer. (pytorchTensorBase.h, raxistensor.h) - The
TensorImpldescribes how to read the numbers: the shape; the strides, how far apart in memory two neighbouring elements of each dimension are; the offset, where in the storage its first number is; and the dtype, the type of each number (float32,int64, …). Strides and offset are what the next sections are about. (pytorchTensorImpl.h, raxistensor_impl.h) - The
StorageImplowns the numbers: a pointer to the bytes, how many there are, and how to free them. It knows nothing about shapes. (pytorchStorageImpl.h, raxisstorage_impl.h)
Why split the last two? Because of y. Many TensorImpls can read one buffer in different ways, and the buffer is freed only when the last of them goes away, because the StorageImpl is reference-counted too. x[1:3] creates one new TensorImpl with a different shape and offset, and no new storage.
This is the part of Raxis’s TensorImpl that matters for this article:
std::shared_ptr<StorageImpl> storage_;
int64_t storage_offset_ = 0;
DimVector shape_;
DimVector strides_;
Dtype dtype_;
PyTorch has the same fields, and more around them: a set of dispatch keys, a slot for its Python object, autograd metadata. Those serve other parts of the system, and we will get to some of them. But the description of the numbers is exactly this.
3. The type and the device are values
Two of the properties of the first section deserve a closer look, because they are values, decided when the program runs, and in C++ that is not the obvious choice.
The dtype in Raxis is a one-byte enum wrapped in a small class that knows the size of an element and the name of the type (raxisdtype.h, raxisDtype). PyTorch has the same two pieces, with more types: the enum is ScalarType (pytorchScalarType.h), and what the TensorImpl stores (pytorchTensorImpl.h) is a TypeMeta, a 16-bit index into a table that holds the size and the name of each type (pytorchtypeid.h, pytorchtypeid.h). The device is a type and an index (pytorchDevice.h, raxisdevice.h): DeviceType::CUDA with index 0 is cuda:0. A Raxis TensorImpl does not even store its device: it asks its storage, whose DataPtr carries it, as we will see (raxistensor_impl.cpp). PyTorch keeps a copy in the TensorImpl as well (pytorchTensorImpl.h).
If you come from C++, you might wonder why the type is not a template parameter, Tensor<float> and Tensor<int64_t>. The compiler would know the type of every element, and every loop would be typed for free. My first tensor in C++, in 2024, was written that way, and it does not hold up in a library like this one:
- Python has one
Tensortype, and the dtype of a tensor is often known only when the program runs: when you load a checkpoint, or when someone passesdtype=. With a template there would be one Python class per type, and aswitchsomewhere to choose between them anyway. - Operations mix types.
int_tensor + float_tensorreturns a float tensor, decided from the two dtypes at runtime: that is type promotion. With templates, every binary operation would need one overload per pair of types. - Many places hold tensors of different types together: the parameters of a model, a
state_dict, the tensors that autograd saves in its graph. They need one type for all of them, so the template would have to be erased again.
So the type is a value everywhere, and becomes a template parameter only where that pays off: inside the kernel, once per operation. This is how Raxis’s add picks its type (raxisbinary_ops_kernel.cpp):
iter.common_dtype().dispatch_all_types([&]<typename scalar_t>(std::type_identity<scalar_t>) {
cpu_kernel_vec(iter,
[](scalar_t a, scalar_t b) -> scalar_t { return a + b; },
[](Vectorized<scalar_t> a, Vectorized<scalar_t> b) { return a + b; });
});
dispatch_all_types is a switch over the dtype that calls the lambda with each C++ type (raxisdtype.h), so the loop inside is compiled once per type, and each version is fully typed. PyTorch does the same with its AT_DISPATCH macros (pytorchDispatch.h). The switch runs once per call to a + b, not once per element. How type promotion picks the type of the result is in the TensorIterator article, and the device is chosen in a similar way one level up, as the CUDA article shows.
4. From an index to a position
For the rest of the article, let’s use a tensor small enough to see:
x = torch.arange(12).reshape(3, 4)
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
Its dtype is int64, the default for integers, so each number takes 8 bytes and the storage is 96 bytes long. Those rows are only how it is printed. The memory is a flat line of 12 numbers, and I chose values that match their positions in it, so that each number tells you where it lives:
0 1 2 3 4 5 6 7 8 9 10 11
A stride says how far to jump in that line when the index of a dimension grows by one. To go to the next row of x you jump 4 positions, and to the next column you jump 1, so its strides are (4, 1). With that, any element is one sum away:
position = offset + i * strides[0] + j * strides[1]
x[1, 2] = 0 + 1 * 4 + 2 * 1 = 6
With more dimensions there are more terms, and nothing else changes. The number of dimensions does not matter to the formula, which will be very convenient in the next article.
The strides of a new tensor come from its shape, going from the last dimension to the first: the last one is 1, and each of the others is the previous stride times the previous size (raxisshape_utils.cpp). This is called row-major order, because the elements of a row end up next to each other.
And the offset is where the first element is. Strides and offset count elements, not bytes, so this is where the dtype comes in: the pointer to the first element, what PyTorch calls data_ptr, is the start of the storage plus the offset times the size of one element, 8 bytes for x (pytorchTensorImpl.h, raxistensor_impl.cpp). That is why a tensor is really a pointer, a shape and strides.
One detail that will catch you if you read both libraries: PyTorch counts its strides and its offset in elements, and NumPy counts its strides in bytes. The same x has strides (4, 1) in PyTorch and (32, 8) in NumPy: 4 elements of 8 bytes to the next row, and one element of 8 bytes to the next column.
The snippets here are PyTorch’s, and they give the same numbers in Raxis.
5. Views
Now look at what an indexing operation has to do. Nothing in the formula mentions the data, so to read a part of x differently, it is enough to write new numbers in a new TensorImpl. Try it:
- shape
- strides
- offset
- contiguous
The tensor you see
The storage, always the same 12 numbers
| View | Shape | Strides | Offset |
|---|---|---|---|
x | (3, 4) | (4, 1) | 0 |
x[1] | (4,) | (1,) | 4 |
x[1:3] | (2, 4) | (4, 1) | 4 |
x[:, 1] | (3,) | (4,) | 1 |
x[:, ::2] | (3, 2) | (4, 2) | 0 |
x.transpose(0, 1) | (4, 3) | (1, 4) | 0 |
Each of these operations is a few lines of arithmetic:
- Picking one index moves the offset by
index * strideand removes that dimension. - A slice moves the offset to where it starts, makes the dimension shorter and, if it has a step, multiplies the stride by that step.
- A transpose swaps two sizes and their strides. Not a single number moves.
This is the slice in Raxis (raxistensor_shape.cpp), without the checks:
int64_t new_offset = self.storage_offset() + start * old_strides[dim];
DimVector new_shape(old_shape);
DimVector new_strides(old_strides);
new_shape[dim] = new_dim_size;
new_strides[dim] = old_strides[dim] * step;
return make_view(self, new_offset, new_shape, new_strides);
PyTorch does the same arithmetic in pytorchTensorShape.cpp, with more branches around it, and creates the result by going through its dispatcher (the layer that routes every call to the right implementation, which Raxis replaces with direct calls) to as_strided. In Raxis, raxismake_view creates the new TensorImpl directly.
6. A stride of 0
There is one more view, and it deserves its own section. Take a column of x, with shape (3, 1), and expand it:
>>> c = x[:, :1]
>>> e = c.expand(3, 4)
>>> e
tensor([[0, 0, 0, 0],
[4, 4, 4, 4],
[8, 8, 8, 8]])
>>> e.stride()
(4, 0)
That tensor has 12 elements and no memory of its own. Its second stride is 0, so moving along a row moves nowhere: all the columns read the same position. Select x[:, :1].expand(3, 4) in the figure above and you will see it, three cells read four times each.
expand is about nothing more than this: every dimension that grows from size 1 gets a stride of 0 (pytorchExpandUtils.cpp, raxistensor_shape.cpp).
So a repeated dimension is not a special case, it is one more stride, and the formula above does not need to change. Keep this in mind for later: broadcasting, when two tensors of different shapes meet in a + b, comes down to this.
7. Sharing has consequences
If two tensors read the same memory, writing through one changes the other:
>>> x = torch.arange(12).reshape(3, 4)
>>> row = x[1]
>>> row[2] = 99
>>> x
tensor([[ 0, 1, 2, 3],
[ 4, 5, 99, 7],
[ 8, 9, 10, 11]])
You probably knew that one. These may be less expected.
Writing to an expanded tensor writes to all of its copies, because they are not copies:
>>> x = torch.arange(12).reshape(3, 4)
>>> e = x[:, :1].expand(3, 4)
>>> e[0, 2] = 99
>>> e
tensor([[99, 99, 99, 99],
[ 4, 4, 4, 4],
[ 8, 8, 8, 8]])
I asked to change one element and four changed, along with x[0, 0]. An in-place operation over the whole tensor does not let you get that far, because each of those positions would be written several times:
>>> e.add_(1)
RuntimeError: unsupported operation: more than one element of the written-to tensor
refers to a single memory location. Please clone() the tensor before performing the operation.
NumPy takes another path here: np.broadcast_to returns a read-only array, so you cannot make the first mistake either.
A small view keeps the whole storage alive. The storage is freed when nothing refers to it, and a view refers to all of it:
big = torch.zeros(10_000_000) # 40 MB
head = big[:2]
del big # the 40 MB are still there
Worse, torch.save writes storages, not views. Saving that head of 2 numbers produces a file of 40 MB, and saving head.clone() a file of 1.6 KB. If you keep a small piece of something big, clone it.
Overwriting a view can break autograd. This one will matter when we get to autograd. To compute gradients, autograd saves some tensors during the forward pass. exp, for example, saves its result, because the derivative of exp(x) is exp(x) itself. If something overwrites those values before the backward pass, the gradient would be wrong, and with views that something does not have to be the saved tensor:
>>> a = torch.ones(3, requires_grad=True)
>>> b = a.exp() # exp saves b for the backward pass
>>> b[:1].mul_(2) # a view of b writes into the memory of b
>>> b.sum().backward()
RuntimeError: one of the variables needed for gradient computation has been modified
by an inplace operation: [torch.FloatTensor [3]], which is output 0 of Exp, is at
version 1; expected version 0 instead. ...
PyTorch catches it with a version counter. Every in-place operation increments it, autograd writes down the version of each tensor it saves, and refuses to run the backward pass if the version has changed since. For this to work, a view and its base must share the same counter, or writing through b[:1] would not change the version of b. PyTorch keeps the counter in the TensorImpl and gives every view the same one as its base (pytorchTensorImpl.h). Raxis keeps it in the StorageImpl (raxisstorage_impl.h), which every view already shares. The rest of the story, how autograd saves and checks, belongs to the autograd articles.
8. Who frees the memory
So a storage lives as long as something refers to it. When the last reference finally goes, who frees its memory, and how? It is less obvious than it looks, because the memory of a tensor comes from very different places, and each one has to be given back in its own way.
The memory comes from an allocator, the piece of the library that hands out memory and takes it back:
- A normal CPU tensor gets it from the CPU allocator, which in the end asks the operating system, and it is freed with the allocator’s
free. - A GPU tensor gets it from the GPU’s caching allocator. Asking the GPU driver for memory (
cudaMalloc) and giving it back (cudaFree) is slow, andcudaFreealso waits for everything the GPU is doing. So when a GPU tensor is freed, its memory is not returned to the driver: it goes back to a cache inside PyTorch, to be handed to the next tensor that needs a block of a similar size. - Pinned memory is CPU memory that the operating system promises to keep in place: it will not move it or swap it to disk. The GPU copies data with its own engine (DMA), which reads the physical memory directly, so it can only copy from memory that stays where it is. From normal memory, the driver first has the CPU copy the data into a pinned buffer of its own, so the CPU is busy with the copy too. From pinned memory there is no intermediate copy, and with
non_blocking=Truethe CPU does not wait: it goes on preparing the next batch while the GPU receives this one. That is why data loaders havepin_memory=True. Allocating it is slow too, so it also comes from a cache of its own. torch.from_numpy(a)does not allocate anything. The tensor reads the memory of the NumPy array, and that memory belongs to NumPy, so the tensor must never free it. What the tensor does is hold a reference to the array, which keeps the array, and its memory, alive while the tensor exists. When the tensor goes away, it drops that reference and nothing more: NumPy frees the memory later, when no one else is using the array.
So the storage keeps, next to the pointer, what has to be released and how. That is PyTorch’s DataPtr (pytorchAllocator.h, raxisdata_ptr.h), built on a UniqueVoidPtr (pytorchUniqueVoidPtr.h, raxisunique_void_ptr.h):
class UniqueVoidPtr {
void* data_; // where the numbers start
std::unique_ptr<void, DeleterFnPtr> ctx_; // what to release, and the function that does it
};
class DataPtr {
UniqueVoidPtr ptr_;
Device device_; // where that memory lives: CPU, cuda:0, ...
};
The data and the context are separate because they are not always the same thing:
- CPU allocator. They are the same pointer, and the deleter frees it (raxis
cpu_allocator.cpp). - Pinned memory. The context is the allocator’s record of the block, and the deleter puts the block back in its cache (raxis
cuda_host_allocator.cpp). - NumPy. The data is the array’s memory and the context is a reference to the array. The deleter only drops that reference, and NumPy frees its memory when nobody else holds the array (pytorch
tensor_numpy.cpp, raxisnumpy_utils.h).
This is the whole chain for torch.from_numpy(a), from the tensor down to the two pointers:
torch.from_numpy(a) builds. The data pointer goes to the numbers, and the context to a reference to the array, so freeing the tensor only drops that reference. With the CPU allocator, both point to the same block, and the deleter frees it.This is what lets one kind of tensor hold memory from anywhere. The StorageImpl never asks where its memory came from: when the last reference to it goes away, the unique_ptr calls the deleter it was given, and the right owner gets its memory back. And because the device travels with the pointer, nothing reads a GPU address as if it were CPU memory. How each device picks its allocator is in the CUDA article.
9. What contiguous really means
is_contiguous() looks easy, and it is easy to read too much into it. Its test is only this (pytorchContiguity.h, raxisshape_utils.cpp):
int64_t expected_stride = 1;
for (int i = ndim - 1; i >= 0; --i) {
if (shape[i] == 1) continue; // never advances, so its stride does not matter
if (strides[i] != expected_stride) return false;
expected_stride *= shape[i];
}
return true;
In words: walking the tensor in its logical order means walking memory one position at a time, with no jumps. Look at what it does not say:
| Tensor | Contiguous | Why |
|---|---|---|
x[1:3] | Yes | It starts at offset 4 and leaves part of the storage out, and that does not matter |
x[1:2], shape (1, 4) | Yes | The size 1 dimension does not count |
x[:, 1:3] | No | After each row, it has to jump over the other columns |
x.transpose(0, 1) | No | It covers its block of memory with nothing left out, but in another order |
x[:, :1], shape (3, 1) | No | Only one column, and each row is 4 positions away |
So contiguous does not mean “the whole storage”, nor “starts at the beginning”, nor even “its numbers fill a block of memory with nothing in between”: the transpose does that and is still not contiguous. It is a statement about order. Any tensor can be walked with its pointer and its strides, but a contiguous one is also a plain array: its numbers are p[0] to p[numel - 1], one loop with one index, which the compiler can vectorize and a copy can do with a single memcpy. Any other tensor needs the shape and strides of every dimension to find the next number, and how to do that fast is the subject of the next article.
And it is one order in particular: row-major, also called C order, the default in PyTorch and NumPy. Fortran stores matrices the other way, column by column, and so do BLAS and LAPACK, which come from it. That is column-major, or Fortran order. NumPy keeps track of both (a.flags['C_CONTIGUOUS'] and a.flags['F_CONTIGUOUS']), while PyTorch’s is_contiguous() checks only the first. The transpose of a row-major matrix is a column-major one. That is why x.transpose(0, 1) is not contiguous, and also why a matrix multiplication can give it to BLAS as it is, telling BLAS that the matrix is transposed instead of copying it (pytorchLinearAlgebra.cpp, raxislinear_algebra.cpp).
When something does need that order, x.contiguous() returns the same tensor if it already has it, without copying, and a copy in the right order otherwise (raxistensor.cpp). That second case is the first time in this article that a number actually moves.
10. view and reshape
These two look like synonyms, and the difference comes from all of the above. view never copies, so it fails if the result cannot be written with strides. reshape tries the same thing and, if it cannot, copies:
>>> t = x.transpose(0, 1) # shape (4, 3), strides (1, 4)
>>> t.view(12)
RuntimeError: view size is not compatible with input tensor's size and stride
(at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
>>> t.reshape(12)
tensor([ 0, 4, 8, 1, 5, 9, 2, 6, 10, 3, 7, 11])
So reshape gives you a tensor that sometimes shares memory with the original and sometimes does not, depending on the strides. If you are going to write to the result, that is worth remembering.
It may surprise you that view does not need a contiguous tensor. It only needs each new dimension to fit in the strides:
>>> x[:, ::2].view(6) # not contiguous, and it works
tensor([ 0, 2, 4, 6, 8, 10])
>>> t.view(2, 2, 3).stride() # neither is this
(2, 1, 4)
x[:, ::2] has shape (3, 2) and strides (4, 2). Its elements are evenly spaced, two positions apart, because the outer stride of 4 is exactly the inner stride of 2 times the inner size of 2, so the two dimensions can merge into one with stride 2. In t, that check fails: merging needs an outer stride of 4 * 3 = 12, and it is 1. Splitting a dimension, as in t.view(2, 2, 3), always works. This is what computeStride works out in pytorchTensorUtils.cpp, and raxiscompute_view_strides in Raxis.
A last limit, in case you come from NumPy: x[::-1] is a negative stride, and PyTorch does not have them. NumPy gives you strides (-32, 8) with no copy, and PyTorch raises an error (its slice has a TODO: support negative strides), so reversing a tensor goes through flip, which copies.
11. How much does it cost?
This is all measured with raxisbench_views.py, on a Ryzen 7 9700X with 8 threads, in November 2026, with PyTorch 2.14.1 and Raxis 0.5.0, on Windows 11 25H2, on float32 square tensors of three sizes:
| Operation | 32 x 32 | 1000 x 1000 | 10000 x 10000 |
|---|---|---|---|
x[1:3], Raxis | 1.2 us | 1.1 us | 1.1 us |
x[1:3], PyTorch | 1.1 us | 1.1 us | 1.5 us |
x.transpose(0, 1), Raxis | 0.6 us | 0.9 us | 0.7 us |
x.transpose(0, 1), PyTorch | 0.9 us | 0.9 us | 0.9 us |
x[:, :1].expand(n, n), Raxis | 2.5 us | 2.6 us | 2.5 us |
x[:, :1].expand(n, n), PyTorch | 2.4 us | 2.4 us | 2.4 us |
x.clone(), Raxis | 0.9 us | 16 us | 22.5 ms |
x.clone(), PyTorch | 1.1 us | 16 us | 24.6 ms |
x.transpose(0, 1).contiguous(), Raxis | 1.1 us | 105 us | 37 ms |
x.transpose(0, 1).contiguous(), PyTorch | 1.7 us | 507 us | 97 ms |
The views take the same time at any size, as the formula promised. They are not free: a microsecond is a new TensorImpl and a Python object around it, so a Python loop that takes a view per row pays that for every row. That is one reason why kernels do not work that way. But between 32 x 32 and 10000 x 10000 there are five orders of magnitude in the data, and none in the time.
The last two rows are the same copy done in a bad order, and that costs more than a plain clone(). The gap between the two libraries there has a concrete reason: for two dimensional transposes, PyTorch has a special path in pytorchCopy.cpp that copies by blocks on a single thread, and Raxis goes through its general copy, in parallel. On one thread, at 1000 x 1000, they are much closer: 0.39 ms against 0.51 ms.
12. What Raxis does differently
The model is the same, so the differences are small. Each one has its reasons, and none of them changes anything you have read here:
- The handle. PyTorch uses its own
intrusive_ptr, with the reference count inside the object, so aTensoris one pointer. Raxis usesstd::shared_ptr, so aTensoris two pointers, in exchange for having no smart pointer of its own to read. - Shape and strides. PyTorch packs both in a single container, pytorch
SizesAndStrides.h, with room for 5 dimensions before it allocates. Raxis has twoDimVectors, which also hold 5 before allocating. In both cases, a view of up to 5 dimensions does not allocate anything else for them. - The version counter. It lives in the
StorageImplin Raxis, instead of being handed around fromTensorImpltoTensorImpl. The cost is that there is nox.datathat opts out of it.
Where this leaves us
We can now describe any tensor with a few numbers, and we know how those numbers change without touching the data. But we have paid for that. A stride can be anything, 1, 4 or 0. The number of dimensions is known only at runtime. So how do you write a loop that adds two of these tensors, and how do you make it fast?
That is the question of How PyTorch walks a tensor of any shape. It is the question I most wanted to answer when I started, and I found little written about it in depth. Before it comes an interlude, What the CPU does when your code reads memory, with the hardware background that the rest of the series refers to: caches, SIMD and threads. If you know it already, you can skip it, or come back to it whenever an article links there.
Further reading
- Edward Yang, “PyTorch internals”, which explains strides with clear drawings, and then goes on to the rest of the system.
- PyTorch’s documentation on tensor views, with the list of operations that return one.
- NumPy’s documentation on the memory layout of an ndarray, the same idea, from the library that popularized it.
If something is not clear, or you think I got something wrong, leave a comment.