Flashcards
DLDM 2022

Flashcards do conjunto (68)
- 1What is a transformer ?A transformer is a Neural network architecture that was introduced for natural language processing. Its key mechanism is the attention layer, that learns what parts of the input sequence are important to understand each token.
- 2How does the Attention Mechanism work ?For every token in the sequence 3 vectors are predicted: Query, Key and Value.
For every token in the sequence the dot product of its Q and every other K vector are computed and "normalized" with softmax.
The softmax-values are then used as weights in a weighted sum of all V vecotrs.
For every token in the input the attention layer outputs 1 vector that is the weighted sum of all V vectors, weighted according to a compatibility function of the Q and K vectors. - 3What is the intuition of attention layers?The attention layer learns which parts of the input are important for understanding a single token in the squence.
A high dot product of Q and K vectors means the token corresponding to K is important to understand the word corresponding to Q.
The V vectors can be interpreted as another embedding of each token and the weihgted sum therefore represents the context for understanding a token. - 4What is the structure of an attention block?Self-Attention layer
(Skip connections)
Feed forward layer(position wise applying the same weihgts to teh output of the SA layer)
(Skip connections ) - 5What is the difference between self- and Crossattention?In self attention the tokens can only attend to the other tokens in the input. In Cross attention the Value vectors are given by the output of another sequence /or input.
CA is used in the Decoder Transformer the V vectors are then provided by the last layer of the encoder. Another application is in image grounded encoder/decoder of BLIP, that allows the transformer to attend to the ouput of an image encoder in order to compute its output. - 6What is BERT?BERT is a bidirectional transformer for anguage understanding.
It is an encoder only transformer that excells at tasks such as sequence classification. - 7How is BERT pretrained?2 Tasks:
Masked token prediction: in the input sequence a number of tokens is masked the task is to predict the real token. The mask is given by either 1. the [MSK] token, 2. a random other token, 3.the actual token. This is done to not bias the resulting model as the MSK token does not exist in any real dataset.
2. Next sentence prediction:
The Model gets two sentences as input seperated by a special token. The task is to predict whether the second sentence is the real continuation of the first. - 8What is the prupose of the [CLS] token?This serves as a token that encodes information from the whole sequence. Its embedding that is computed by the transformer can be used as input to another MLP for sequence level classfication tasks.
- 9How many parameters does BERT have?~340 million
24 layers
16 multihead attention heads
hidden size of 1024 - 10What is multihead attention?In order to be able to learn multiple contexts, multiple attention heads are used in paralell in the same layer. Their output is concatenated and recombined into a single embedding using a learned linear projection.
This is simialr to using multiple Convolution kernels in one Conv layer. It simply allows for discovering more patterns at each level of encoding. - 11What is a visual transformer?A visual transformer is applying the transformer architecture to image data.
- 12What preprocessing is necessary to feed images into the transformer?Pixel level attention would be too costly to compute therefore the image is split into patches. These patches are then flattened and turned int N-dimensional vectors that are encoded according to their position in the sequence. For classification a [CLS] token is prepended and the whole is input to the transformer.
- 13What is the architecture of a Visual transformer?The preprocessed input sequence is input to a encoder only transformer. The embeddign that corresponds to the [CLS] token is used as input into a MLP, that predicts the class of the image.
- 14What is Word2Vec?Word2Vec is a method to learn word embeddings that encode some similarity of words. Unlike in a one-hot encoding similar words have similar vector representation, that make even some form of arithmetics possible such as Queen - Woman + Man = King
- 15How does Word2Vec work ?two approaches, that work in opposite direction. A single hidden layer NN is used to predict the context(words before and after a target word) from a target word(Skip gram) or the target word from the context (CBOW)
The initial encoding are the one hot encoded words from a vocabulary.
Once the model is trained the one hot encoded words can be encoded using the input weights to the NN. - 16What is an LSTM?LSTMs are an extension to regualr RNN. RNNs have the problem of vanishing gradients, due to the recursive formulation of their gradients, which makes it impossible for them to learn long term dependencies. LSTMs solve this probelm using some improvements.
- 17What is the forget gate?The forget gate controls what parts of the Cell state should be forgotten.
The input is the concatented hidden state from the previous time step and the newest input. This is fed into a Sigmoid layer with output dimensionality equals the size of the cell state. The result is multiplied with the cell state, thus removing parts that should be forgotten - 18What is the input gate?The input gate controls which parts of the newest input should be added to the cell state and to which extend.
First the concatenation of last hidden state and newest input is fed into a sigmoid and a tanh layer. These two outputs are multiplied and then added to the Cell state. The tanH layer controls what should be added to Ct and the sigmoid layer controls the extent. - 19What is the output gate?The output gate computes the next hidden state (that also serves as output) from the current Cell state and input/last hidden state.
The Input /las thidden state is fed into a sigmoid layer while the Cell state (already manipulated by add and forget gate) is input to a tanh layer. The two outputs are combined by multiplying to form the next hidden state. - 20What is UMAP ?UMAP is a dimension reduction algorithm that consists of two steps:
1. The input data is used build a graph
2. A low dimensional mapping is computed that results in a structurally similar graph
UMAP is mainly practica for visualization as the directions, cluster size or axes in the low dimensional size do not have any meaning
UMAP tends to be able to make global structure visible but the local/global tradeoff can be controlled using two parameters - 21What are the two parameters in UMAP?Number of neighbours: Controls the number of neighbours that are considered when building the high dimensional graph
high values will keep global structure
low values will keep local structure
Minimum distance:The minimal distance two points can have in the low diemsnional space.
high values will result in a mapping that focuses on global structure
low values will keep more local structures - 22What are the rules for building the graph in UMAP?1. Edge weights represent the likelihood that connected nodes are indeed connected.
2. Every node must at least be connected to its nearest neighbour
3. Grow circles around nodes; when circles overlap add edge between the nodes; the weight is Antiproportional to their distance
4. Maximum Radius of circles is not a global parameter but is instead computed from the n-nearest neighbours of each node. - 23How is the low dimensional mapping computed using the Graph?Goal is to find a mapping that results in a similar graph when applying the same rules.
This is done using force-directed graph layout algorithm, that applies attractive forces on nodes along edges and repulsive ones between nodes. The forces are derived from the gradient wrt to point coordinates, that maximizes the cross entropy of edge weights of high and low dim graphs. - 24What is CLIP?CLIP simultanesously trains a image and a text encoder using a contrastive learning task. This leads to similar embeddings for text and images if text is a good caption for the image.
- 25How does the Contrastive learning work in CLIP ?Dataset constist of a set of image-text pairs. The text encoder and the image encoder compute their embeddings. If the pair is matched in the dataset the cosine similarity of image- and text embedding should be high -> maximize wrt to the network weights
If the pair is not matched the cosine similarity should be low- > maximize wrt to the network params. - 26How can CLIP embeddings be used for Zero shot learning?Both encoders are trained using the contrastive learning goal.
For classification create artificial captions for each class (such as "image of a [CLASS]")
then feed this caption for every class into the text encoder and the image into the image encoder. Choose the class corresponding to the cpation with highest cosine similarity. - 27What are diffusion models?Diffusion Models are generative networks that work by reversing a diffusion process that iteratively adds gaussian noise to a real sample of data/image. The model is trained to predict the noise that was added at a timestep from the noisy image and the timestep.
New Samples are then generated by the trained model by starting from a random noise iteratively removing the predicted noise until a clear sample is left - 28What are advantages of diffusion models compared to other generateive models?VAEs produce very blurry images due to the photometric loss that tends to be minimal for "mean pixel values"
GANs produce high res images but are difficult to train.
Diffusion models are easy to train and produce high quality images, due to the iterative process in the generation it does take a while to generate new images. - 29How does the training of a Diffusion model look like?1. Take an image from the training set and sample a number between 1 and T(the maximum diffusion step)
2. Compute the noised image (can be done in a single computation)
3. Input the timestep and the noised image into the model
4. compute MSE between real noise and the predicted noise - 30What is the model architecture of a Diffuson model?This depends on the type of data that should be generated.
For image data a UNet Architecture is suitable since the size of the latent varaible is the same as the output size. - 31What is DallE( 2) ?Dalle2 uses the image embeddings of learned by CLIP for generating images for a given prompt. The main advantage of using the CLIP image embeddings is the easy image manipulation based on textual clues. Since CLIP embeds text and images into the same space, directions given by a certain prompt can be used to shift the embedding an image was generated from and thus manipulating it according to the clues in the prompt. In GANs it takes a lot of work to find these directions.
- 32What is the purpose of the prior model in Dalle2?Dall e uses the image embeddings of CLIP to generate new images. To get the image embeddings that belong to a given text a model is needed that "inverts the CLIP model", ie predicts image embeddings from text.
The model is a decoder only transformer model that takes as input the 1. Text, 2.the CLIP text embeddings, 3. a timestep embedding for the diffusion, 4.the noised Image embeddings 5. a final embedding whose output is used to predict the image embedding.
instead of predicting the noise as would normally be done in diffusion model the model is trained to directly predict the CLIP image embeddings. - 33What is the Decoder in Dalle 2?The decoder generates the image for a given text prompt by conditioning a diffusion model on the CLIP image embeddings predicted by the prior.
- 34What is the Decoder architecture in Dalle2?The Decoder is a UNet with residual blocks in the down- as well the up sample pathways. At the 8x8, 16x16 and 32x32 resolution levels global attention layers with a single head are introduced.
- 35How does the conditioning of the diffusion decoder work in Dalle?At each residual block: the timestep embedding as well as the CLIP image embeddings are projected to the correct size and added.
Additionally the text prompt is fed into an additional text encoder whose output is also added after the residual blocks
CLIP embeddings are projected into 4 tokens that are added to this text encoder output. This is projected to the correct size and added to the context predictions of the global attention layers. - 36What is guidance in diffusion models and how is it used in DAlle2?Guidance is the process of perturbing samples of a conditional diffusion models by the gradient of the log probability of the class of the sampe. This improves the sample quality. In Dalle2 they use classfier free guidance which shifts the output of the diffusion model towards that of the desired class and away from the empty set.
To do this the conditioning on the empty set must be learned by setting te conditioning embeddings to zero with some probability during training. The shifting then only happens during sampling - 37What is the pretraining task of GPT-3?GPT (Generative Pre Training)
The task consists of predicting the next token from the previous ones in the sequence. - 38What is the architecture of GPT-3?The general architecture is given by an decoder-only transformer. Since there is no encoder the cross attention layer is dropped, which means the only difference to an encoder-only architecture such as BERT is the unidirectional/causal attention mechanism in the self attention layers; meaning token can only attend to tokens to their left.
Some minor changes to the original architecture of Vasvani et al:
1. Layer norm was moved to the input of each Attention sub block as well as after the last layer of the last attention block - 39How many parameters does GPT-3 have?175 billion
96 transformer blocks
96 multihead heads
hiddensize: 12288
orders of magnitude bigger than the biggest bert model with 340 million params - 40What is few-shot Learning for GPT-3?Using the pretrained GPT3 model to solve tasks it was not trained for without any weight updates, by only feeding example tasks into its context window.
Example Question answering: Feed into model "Q1 A1 Q2 A2 ... QN AN Qtest "
Correct output would be Atest
For multiple choice only feed the correct answer and then evaluate the likelihood for every possible answer for the test question. - 41What is Video pretraining (for Minecraft)?VPT tries to leverage the huge amounts of videos of players playing minecraft to train an agent in this environment. Since the data is unlabeled (actions taken are unknown) first an inverse dynamics model is trained that predicts the actions taken in a given video sequence. This IDM is then used to label the video data which is then used to train a Behavioural Cloning agent.
- 42What is the action space in VPT for minecraft?The actions that the agent takes arer the same ones a human player would take. Ie predict individual keypresses and mouse movements instead of meta actions that consists of multiple low level actions.
- 43How does Finetuning work in VPT?The base model trained solely using the aritifivcially annotated data already performs quiet well. To finetune it to solve specific tasks, the videos to train the model on are chosen by a given prompt. These videos should contain the desired tasks.
Further finetuning can be done using Reinforcement Learning: this order achieved the best results - 44How and why are the videos in VPT filtered?Some videos contain watermarks or overlayed videos of the player. The videos are filtered to contain only the elements the normal MC env would have. For this a small set of labeled data is used to train a small model that predicts for a given frame whether it is clean or not. Only clean frames are classified by the IDM and used for BC
- 45What is the model architecture of the IDM model?Input: stack of 128 frames
1. Conv3d layer that temporally convolves the stack
2. ResNet that ouputs one vector for every frame
3. two Fully connected layers
4. Transformer
5. Embeddings corresponding to each frame are added into individual output heads for every low level action:
binary output for every keypress
11 class output for the two discretized mouse directions - 46What is the model architecture for the Behaviuoural clining model?The same as the IDM model with the difference that it is causal as only previous frames can be used to predict the next action. To do thsi the conv3d layer is dropped and the SelfAttention layers in the transformer are changed to Masked Self attention
- 47How many parameters does the VPT models have?The IDM and Behavioural cloning models have ~ 0.5billion parameters.
For the Reinforcement Learning models with half the size were used. This was achieved by reducing the width of the layers. - 48What is NerF?Neural Radiance Fields is a neural network for rendering views from new camera poses for a given scene. This is done using simple fully connected networks, that predict the color and the density for point in the scene (X,Y,Z coordinates + viewing direction).
- 49How is a new view generated using NeRF?1. Pass rays through the scene from the desired camera pose (4096)
2. Sample points along these rays (64)
3. Feed points into NeRF to compute RGB & density values
4. Use classical volume based rendering to synthesize a new view from the networks output - 50What 2 improvements were added to the basic NeRF model?1. The 5D input for the model is mapped into a high dimensional space in order to be able to capture high frequency feature in the generated images. Every component of the 5D input is mapped into a 2L dimensional vector using a series of sine and cosine functions with decreasing frequency from (2^0 * pi) to (2^(L-1) *pi)
2.Hierarchical Sampling of points along the rays. The simple sampling technique samples points that may not be visible to the camera. The goal is to sample visible points with a higher probability. This is done using two networks simultaneously one fine and one coarse. The coarse network is used to evaluate the points sampled by stratified sampling along each ray. The output of the coarse network is used to define a PDF along the ray. In a second step points are sampled according to this PDF and the union of the first and second set are used as input to the fine network that produces the final RGB and density values - 51What is the density that is predicted by the NeRF?The density can be interpreted as the probability that a ray terminates at this point.
- 52What is the volume based rendering technique used in NeRF?Essentially the RGB values of points along a ray are accumulated and weighted according to the points density.
In NeRF this is discretized by using randomly sampleld points along the ray, which then boils down to alpha compositing where the alpha values are computed from the density predicted by teh NeRF. - 53How is NeRF trained?To train NeRF a dataset with known camera poses is needed. The training then simply constists of generating views and minimizing the MSE of generated and target image.
- 54What are advantages of NeRF?1. Very simple model structure and therefore fewer weights than conv based models
2. No specific restrictions on the scenes that can be learned (some models require a specific background)
3. Produces high quality images. - 55What is the model architecture of NeRF?High frequency encoded Point coordintaes -> 8 Fully connected layers with ReLU activation and 256 nodes --> output 1: density ; output 2: 256 dim feature vector
Feature vector + encoding of Viewing direction -> 128 node FC layer with ReLU activation-> RGB output
The density is independent on the viewing angle which is why it is fed in later in the network to only condition the RGB values - 56What is BLIP?BLIP is a image-language learning framework, with two contributions: the Multimodal mixture encoder-decoder,
a language image model excelling both at image text retrieval as well as captioning
and CaptFilt a bottstrapping mecahnism to filter noisy web data that is usually used for training of image language models - 57What is the MED?The MEd consists of :
1. An Image encoder that is a visual transformer
2. an image grounded encoder (encoder only transformer)
3. An text encoder (encoder only transformer)
4. An image grounded text decoder (decoder only transformer) - 58How is the MED trained?There are three loss functions to train the MED for both image text retrieval and captioning.
1. The ITC (image text contrastive) loss; the same loss functions as CLIP for optimizing the image encoder and the text encoder
2. The ITM (image-text matching) loss for optimizing the image grounded encoder and the image encoder for binary classification whether an given image text pair is matched in the dataset
3. The language modeling loss (generative training procedure) to optimize the image encoder and image grounded text decoder, to generate fitting cpations for an image - 59What is the model architecture of the MED?The image encoder is a visual transformer.
The Text models share most of their weights. Feed forward layers in attention blocks are shared across all three text models
The Cross attention layers of the image grounded models are shared
The bidirectional Self attention layers of the the two text encoders are shared - 60What is CaptFilt?CaptFilt is a method to filter the noisy web data for image language learning by using a small clean dataset (COCO)
1. Pre train an MED using The noisy data
2. Finetune the individual models on their respective loss individually on the COCO dataset
3. Use MED to identify bad samples from a noisy dataset; Produce new Captions
4. Use the filtered dataset to train a new MED from scratch
5. repeat - 61What is the loss function in CLIP?CLIP uses Cross entropy loss in order to maximize the cosine similarity of matched and minimize it for unmatched pairs.
- 62What is CLIP4Caption?CLIP4Caption is a learning framework that trains a model to generate captions for a given video sequence.
- 63How does the training in CLIP4Caption work?The training is split into two phases:
1. The pre training phase that trains a video encoder and a text encoder using the contrastive learning objective of CLIP
2. A finetuning phase training a Video Encoder and Language Decoder to generate captions from the video embeddings learned in the first phase. - 64Describe the Pretraining phase of CLIP4Caption in more detail!The video sequence is split into K splits, from each 1 random frame is selected. These K frames are input into a visaul Transformer that computes one embedding for each frame. These embeddings are combined using Mean Pooling.
A Text encoder gets the captions as input and computes a embedding for the text.
On the Mean pooled video embedding and the text embedding the contrastive learning objective is applied minimizing the Cosine similarity of unmatched pairs and maximizing it for matched pairs using a Cross entropy Loss.
Both encoders are initialized to the pretrained weights of a CLIP image text retrieval model - 65Describe the Finetuning Phase of CLIP4Caption in more detail!In the finetuning phase a transformer based Encoder-Decoder model is trained to generate captions for a given video.
The Encoder gets as input the Video embeddings (not pooled) from the encoder trained in the pretraining phase.
The decoder then computes a sequence of tokens.
The model is trained using a standard Language modeling loss that is essentially the sum of cross entropies between ouput probabilities over the vocabulary and the one hot encoded next token, over the whole output sequence. - 66What is the model architecture of the Video and Text encoder in CLIP4Caption?The image encoder is a visual transformer with 12 layers. The patches the images are divided in has a size of 32
The text encoder is a 12 layer transformer model with a hidden size of 512 - 67What is the model architecture of the Encoder-Decoder transformer in CLIP4Caption?The Encoder is a small 1-layer transformer, since it works on the already video embeddings it only needs to enhance these feature a little.
The Decoder is a 3 layer decoder transformer. - 68What is the formula for the Cross Entropy loss?M- number of classes
y- The binary label for the sample
p - the predicted proabability over classes of the model
Since all ys except for the correct one are zero this boils down to:
- log(p(y))
Informações
- Descrição
- FlashCards sobre DLDM 2022, criado por Laurenz Nippelbeißer em 03-09-2022.
- Criado por
- Laurenz Nippelbeißer
- Visualizações
- 2
- Avaliação